With the artificial intelligence conversation now mainstream, the 2023 MIT-MGB AI Cures conference saw attendance double from previous years.
( 8
min )
AWS delivers services that meet customers’ artificial intelligence (AI) and machine learning (ML) needs with services ranging from custom hardware like AWS Trainium and AWS Inferentia to generative AI foundation models (FMs) on Amazon Bedrock. In February 2022, AWS and Hugging Face announced a collaboration to make generative AI more accessible and cost efficient. Generative […]
( 7
min )
This post is co-written with Thatcher Thornberry from bpx energy. Facies classification is the process of segmenting lithologic formations from geologic data at the wellbore location. During drilling, wireline logs are obtained, which have depth-dependent geologic information. Geologists are deployed to analyze this log data and determine depth ranges for potential facies of interest from […]
( 11
min )
An update to the Omniverse Connector for Adobe Substance 3D Painter will save 3D creators across industries significant time and effort.
( 6
min )
In my last project, I had to come up with some code and algorithm to solve an interesting problem. I realized that it could lead to some off-the-beaten-path job interview question. The problem is a fundamental one. The level ranges from elementary school to one of the most difficult unsolved problems of all times, depending… Read More »An Intriguing Job Interview Question for AI/ML Professionals
The post An Intriguing Job Interview Question for AI/ML Professionals appeared first on Data Science Central.
( 21
min )
NASA's Kepler Space Telescope has been instrumental in the task of finding
the presence of exoplanets in our galaxy. This search has been supported by
computational data analysis to identify exoplanets from the signals received by
the Kepler telescope. In this paper, we consider building upon some existing
work on exoplanet identification using residual networks for the data of the
Kepler space telescope and its extended mission K2. This paper aims to explore
how deep learning algorithms can help in classifying the presence of exoplanets
with less amount of data in one case and a more extensive variety of data in
another. In addition to the standard CNN-based method, we propose a Siamese
architecture that is particularly useful in addressing classification in a
low-data scenario. The CNN and ResNet algorithms achieved an average accuracy
of 68% for three classes and 86% for two-class classification. However, for
both the three and two classes, the Siamese algorithm achieved 99% accuracy.
( 2
min )
D-Adaptation is an approach to automatically setting the learning rate which
asymptotically achieves the optimal rate of convergence for minimizing convex
Lipschitz functions, with no back-tracking or line searches, and no additional
function value or gradient evaluations per step. Our approach is the first
hyper-parameter free method for this class without additional multiplicative
log factors in the convergence rate. We present extensive experiments for SGD
and Adam variants of our method, where the method automatically matches
hand-tuned learning rates across more than a dozen diverse machine learning
problems, including large-scale vision and language problems.
An open-source implementation is available.
( 2
min )
Deep Learning (DL) has the potential to optimize machine learning in both the
scientific and clinical communities. However, greater expertise is required to
develop DL algorithms, and the variability of implementations hinders their
reproducibility, translation, and deployment. Here we present the
community-driven Generally Nuanced Deep Learning Framework (GaNDLF), with the
goal of lowering these barriers. GaNDLF makes the mechanism of DL development,
training, and inference more stable, reproducible, interpretable, and scalable,
without requiring an extensive technical background. GaNDLF aims to provide an
end-to-end solution for all DL-related tasks in computational precision
medicine. We demonstrate the ability of GaNDLF to analyze both radiology and
histology images, with built-in support for k-fold cross-validation, data
augmentation, multiple modalities and output classes. Our quantitative
performance evaluation on numerous use cases, anatomies, and computational
tasks supports GaNDLF as a robust application framework for deployment in
clinical workflows.
( 3
min )
Over-approximating the reachable sets of dynamical systems is a fundamental
problem in safety verification and robust control synthesis. The representation
of these sets is a key factor that affects the computational complexity and the
approximation error. In this paper, we develop a new approach for
over-approximating the reachable sets of neural network dynamical systems using
adaptive template polytopes. We use the singular value decomposition of linear
layers along with the shape of the activation functions to adapt the geometry
of the polytopes at each time step to the geometry of the true reachable sets.
We then propose a branch-and-bound method to compute accurate
over-approximations of the reachable sets by the inferred templates. We
illustrate the utility of the proposed approach in the reachability analysis of
linear systems driven by neural network controllers.
( 2
min )
A common pipeline in learning-based control is to iteratively estimate a
model of system dynamics, and apply a trajectory optimization algorithm -
e.g.~$\mathtt{iLQR}$ - on the learned model to minimize a target cost. This
paper conducts a rigorous analysis of a simplified variant of this strategy for
general nonlinear systems. We analyze an algorithm which iterates between
estimating local linear models of nonlinear system dynamics and performing
$\mathtt{iLQR}$-like policy updates. We demonstrate that this algorithm attains
sample complexity polynomial in relevant problem parameters, and, by
synthesizing locally stabilizing gains, overcomes exponential dependence in
problem horizon. Experimental results validate the performance of our
algorithm, and compare to natural deep-learning baselines.
( 2
min )
While Generative Adversarial Networks (GANs) have recently found applications
in image editing, most previous GAN-based image editing methods require
largescale datasets with semantic segmentation annotations for training, only
provide high level control, or merely interpolate between different images.
Previous researchers have proposed EditGAN for high-quality, high-precision
semantic image editing with limited semantic annotations by finding `editing
vectors'. However, it is noticed that there are many features that are not
highly associated with semantics, and EditGAN may fail on them. Based on the
orthogonality of latent space observed by EditGAN, we propose a method to
estimate editing vectors that do not rely on semantic segmentation nor
differentiable feature estimation network. Our method assumes that there is a
correlation between the intensity distribution of features and the distribution
of hidden vectors, and estimates the relationship between the above
distributions by sampling the feature intensity of the image corresponding to
several hidden vectors. We modified Linear Discriminant Analysis (LDA) to deal
with both binary feature editing and continuous feature editing. We then found
that this method has a good effect in processing features such as clothing type
and texture, skin color and hair.
( 2
min )
The article presents the torchosr package - a Python package compatible with
PyTorch library - offering tools and methods dedicated to Open Set Recognition
in Deep Neural Networks. The package offers two state-of-the-art methods in the
field, a set of functions for handling base sets and generation of derived sets
for the Open Set Recognition task (where some classes are considered unknown
and used only in the testing process) and additional tools to handle datasets
and methods. The main goal of the package proposal is to simplify and promote
the correct experimental evaluation, where experiments are carried out on a
large number of derivative sets with various Openness and class-to-category
assignments. The authors hope that state-of-the-art methods available in the
package will become a source of a correct and open-source implementation of the
relevant solutions in the domain.
( 2
min )
Dynamical mean-field theory is a powerful physics tool used to analyze the
typical behavior of neural networks, where neurons can be recurrently
connected, or multiple layers of neurons can be stacked. However, it is not
easy for beginners to access the essence of this tool and the underlying
physics. Here, we give a pedagogical introduction of this method in a
particular example of generic random neural networks, where neurons are
randomly and fully connected by correlated synapses and therefore the network
exhibits rich emergent collective dynamics. We also review related past and
recent important works applying this tool. In addition, a physically
transparent and alternative method, namely the dynamical cavity method, is also
introduced to derive exactly the same results. The numerical implementation of
solving the integro-differential mean-field equations is also detailed, with an
illustration of exploring the fluctuation dissipation theorem.
( 2
min )
Denoising diffusion models are a class of generative models which have
recently achieved state-of-the-art results across many domains. Gradual noise
is added to the data using a diffusion process, which transforms the data
distribution into a Gaussian. Samples from the generative model are then
obtained by simulating an approximation of the time reversal of this diffusion
initialized by Gaussian samples. Recent research has explored adapting
diffusion models for sampling and inference tasks. In this paper, we leverage
known connections to stochastic control akin to the F\"ollmer drift to extend
established neural network approximation results for the F\"ollmer drift to
denoising diffusion models and samplers.
( 2
min )
Localizing behaviors of neural networks to a subset of the network's
components or a subset of interactions between components is a natural first
step towards analyzing network mechanisms and possible failure modes. Existing
work is often qualitative and ad-hoc, and there is no consensus on the
appropriate way to evaluate localization claims. We introduce path patching, a
technique for expressing and quantitatively testing a natural class of
hypotheses expressing that behaviors are localized to a set of paths. We refine
an explanation of induction heads, characterize a behavior of GPT-2, and open
source a framework for efficiently running similar experiments.
( 2
min )
The concept of neuromuscular activity recognition using instantaneous
high-density surface electromyography (HD-sEMG) images opens up new avenues for
the development of more fluid and natural muscle-computer interfaces. However,
the existing approaches employed a very large deep convolutional neural network
(ConvNet) architecture and complex training schemes for HD-sEMG image
recognition, which requires the network architecture to be pre-trained on a
very large-scale labeled training dataset, as a result, it makes
computationally very expensive. To overcome this problem, we propose S-ConvNet
and All-ConvNet models, a simple yet efficient framework for learning
instantaneous HD-sEMG images from scratch for neuromuscular activity
recognition. Without using any pre-trained models, our proposed S-ConvNet and
All-ConvNet demonstrate very competitive recognition accuracy to the more
complex state of the art for neuromuscular activity recognition based on
instantaneous HD-sEMG images, while using a ~ 12 x smaller dataset and reducing
learning parameters to a large extent. The experimental results proved that the
S-ConvNet and All-ConvNet are highly effective for learning discriminative
features for instantaneous HD-sEMG image recognition especially in the data and
high-end resource constrained scenarios.
( 2
min )
In recent years, there has been a growing interest in the effects of data
poisoning attacks on data-driven control methods. Poisoning attacks are
well-known to the Machine Learning community, which, however, make use of
assumptions, such as cross-sample independence, that in general do not hold for
linear dynamical systems. Consequently, these systems require different attack
and detection methods than those developed for supervised learning problems in
the i.i.d.\ setting. Since most data-driven control algorithms make use of the
least-squares estimator, we study how poisoning impacts the least-squares
estimate through the lens of statistical testing, and question in what way data
poisoning attacks can be detected. We establish under which conditions the set
of models compatible with the data includes the true model of the system, and
we analyze different poisoning strategies for the attacker. On the basis of the
arguments hereby presented, we propose a stealthy data poisoning attack on the
least-squares estimator that can escape classical statistical tests, and
conclude by showing the efficiency of the proposed attack.
( 2
min )
Digital transformation in buildings accumulates massive operational data,
which calls for smart solutions to utilize these data to improve energy
performance. This study has proposed a solution, namely Deep Energy Twin, for
integrating deep learning and digital twins to better understand building
energy use and identify the potential for improving energy efficiency. Ontology
was adopted to create parametric digital twins to provide consistency of data
format across different systems in a building. Based on created digital twins
and collected data, deep learning methods were used for performing data
analytics to identify patterns and provide insights for energy optimization. As
a demonstration, a case study was conducted in a public historic building in
Norrk\"oping, Sweden, to compare the performance of state-of-the-art deep
learning architectures in building energy forecasting.
( 2
min )
In manufacturing settings, data collection and analysis are often a
time-consuming, challenging, and costly process. It also hinders the use of
advanced machine learning and data-driven methods which require a substantial
amount of offline training data to generate good results. It is particularly
challenging for small manufacturers who do not share the resources of a large
enterprise. Recently, with the introduction of the Internet of Things (IoT),
data can be collected in an integrated manner across the factory in real-time,
sent to the cloud for advanced analysis, and used to update the machine
learning model sequentially. Nevertheless, small manufacturers face two
obstacles in reaping the benefits of IoT: they may be unable to afford or
generate enough data to operate a private cloud, and they may be hesitant to
share their raw data with a public cloud. Federated learning (FL) is an
emerging concept of collaborative learning that can help small-scale industries
address these issues and learn from each other without sacrificing their
privacy. It can bring together diverse and geographically dispersed
manufacturers under the same analytics umbrella to create a win-win situation.
However, the widespread adoption of FL across multiple manufacturing
organizations remains a significant challenge. This study aims to review the
challenges and future directions of applying federated learning in the
manufacturing industry, with a specific emphasis on the perspectives of
Industry 4.0 and 5.0.
( 3
min )
In this work we introduce a self-supervised pretraining framework for
transformers on functional Magnetic Resonance Imaging (fMRI) data. First, we
pretrain our architecture on two self-supervised tasks simultaneously to teach
the model a general understanding of the temporal and spatial dynamics of human
auditory cortex during music listening. Our pretraining results are the first
to suggest a synergistic effect of multitask training on fMRI data. Second, we
finetune the pretrained models and train additional fresh models on a
supervised fMRI classification task. We observe significantly improved accuracy
on held-out runs with the finetuned models, which demonstrates the ability of
our pretraining tasks to facilitate transfer learning. This work contributes to
the growing body of literature on transformer architectures for pretraining and
transfer learning with fMRI data, and serves as a proof of concept for our
pretraining tasks and multitask pretraining on fMRI data.
( 2
min )
"Data is the new oil", in short, data would be the essential source of the
ongoing fourth industrial revolution, which has led some commentators to
assimilate too quickly the quantity of data to a source of wealth in itself,
and consider the development of big data as an quasi direct cause of profit.
Human resources management is not escaping this trend, and the accumulation of
large amounts of data on employees is perceived by some entrepreneurs as a
necessary and sufficient condition for the construction of predictive models of
complex work behaviors such as absenteeism or job performance. In fact, the
analogy is somewhat misleading: unlike oil, there are no major issues here
concerning the production of data (whose flows are generated continuously and
at low cost by various information …
( 3
min )
The detection of malicious websites has become a critical issue in
cybersecurity. Therefore, this paper offers a comprehensive review of
data-driven methods for detecting malicious websites. Traditional approaches
and their limitations are discussed, followed by an overview of data-driven
approaches. The paper establishes the data-feature-model-extension pipeline and
the latest research developments of data-driven approaches, including data
preprocessing, feature extraction, model construction and technology extension.
Specifically, this paper compares methods using deep learning models proposed
in recent years. Furthermore, the paper follows the
data-feature-model-extension pipeline to discuss the challenges together with
some future directions of data-driven methods in malicious website detection.
( 2
min )
Backpropagation (BP) is the most important gradient estimation method for
training neural networks in deep learning. However, the literature shows that
neural networks trained by BP are vulnerable to adversarial attacks. We develop
the likelihood ratio (LR) method, a new gradient estimation method, for
training a broad range of neural network architectures, including convolutional
neural networks, recurrent neural networks, graph neural networks, and spiking
neural networks, without recursive gradient computation. We propose three
methods to efficiently reduce the variance of the gradient estimation in the
neural network training process. Our experiments yield numerical results for
training different neural networks on several datasets. All results demonstrate
that the LR method is effective for training various neural networks and
significantly improves the robustness of the neural networks under adversarial
attacks relative to the BP method.
( 2
min )
Neurosymbolic AI deals with models that combine symbolic processing, like
classic AI, and neural networks, as it's a very established area. These models
are emerging as an effort toward Artificial General Intelligence (AGI) by both
exploring an alternative to just increasing datasets' and models' sizes and
combining Learning over the data distribution, Reasoning on prior and learned
knowledge, and by symbiotically using them. This survey investigates research
papers in this area during recent years and brings classification and
comparison between the presented models as well as applications.
( 2
min )
In this work, we propose a deep learning (DL)-based constitutive model for
investigating the cyclic viscoelastic-viscoplastic-damage behavior of
nanoparticle/epoxy nanocomposites with moisture content. For this, a long
short-term memory network is trained using a combined framework of a sampling
technique and a perturbation method. The training framework, along with the
training data generated by an experimentally validated
viscoelastic-viscoplastic model, enables the DL model to accurately capture the
rate-dependent stress-strain relationship and consistent tangent moduli. In
addition, the DL-based constitutive model is implemented into finite element
analysis. Finite element simulations are performed to study the effect of load
rate and moisture content on the force-displacement response of nanoparticle/
epoxy samples. Numerical examples show that the computational efficiency of the
DL model depends on the loading condition and is significantly higher than the
conventional constitutive model. Furthermore, comparing numerical results and
experimental data demonstrates good agreement with different nanoparticle and
moisture contents.
( 2
min )
PiML (read $\pi$-ML, /`pai.`em.`el/) is an integrated and open-access Python
toolbox for interpretable machine learning model development and model
diagnostics. It is designed with machine learning workflows in both low-code
and high-code modes, including data pipeline, model training, model
interpretation and explanation, and model diagnostics and comparison. The
toolbox supports a growing list of interpretable models (e.g. GAM, GAMI-Net,
XGB2) with inherent local and/or global interpretability. It also supports
model-agnostic explainability tools (e.g. PFI, PDP, LIME, SHAP) and a powerful
suite of model-agnostic diagnostics (e.g. weakness, uncertainty, robustness,
fairness). Integration of PiML models and tests to existing MLOps platforms for
quality assurance are enabled by flexible high-code APIs. Furthermore, PiML
toolbox comes with a comprehensive user guide and hands-on examples, including
the applications for model development and validation in banking. The project
is available at https://github.com/SelfExplainML/PiML-Toolbox.
( 2
min )
Recent years have witnessed a growth in mathematics for deep learning--which
seeks a deeper understanding of the concepts of deep learning with mathematics
and explores how to make it more robust--and deep learning for mathematics,
where deep learning algorithms are used to solve problems in mathematics. The
latter has popularised the field of scientific machine learning where deep
learning is applied to problems in scientific computing. Specifically, more and
more neural network architectures have been developed to solve specific classes
of partial differential equations (PDEs). Such methods exploit properties that
are inherent to PDEs and thus solve the PDEs better than standard feed-forward
neural networks, recurrent neural networks, or convolutional neural networks.
This has had a great impact in the area of mathematical modeling where
parametric PDEs are widely used to model most natural and physical processes
arising in science and engineering. In this work, we review such methods as
well as their extensions for parametric studies and for solving the related
inverse problems. We equally proceed to show their relevance in some industrial
applications.
( 2
min )
We characterise the learning of a mixture of two clouds of data points with
generic centroids via empirical risk minimisation in the high dimensional
regime, under the assumptions of generic convex loss and convex regularisation.
Each cloud of data points is obtained by sampling from a possibly uncountable
superposition of Gaussian distributions, whose variance has a generic
probability density $\varrho$. Our analysis covers therefore a large family of
data distributions, including the case of power-law-tailed distributions with
no covariance. We study the generalisation performance of the obtained
estimator, we analyse the role of regularisation, and the dependence of the
separability transition on the distribution scale parameters.
( 2
min )
This work concerns controlling the false discovery rate (FDR) in networks
under communication constraints. We present sample-and-forward, a flexible and
communication-efficient version of the Benjamini-Hochberg (BH) procedure for
multihop networks with general topologies. Our method evidences that the nodes
in a network do not need to communicate p-values to each other to achieve a
decent statistical power under the global FDR control constraint. Consider a
network with a total of $m$ p-values, our method consists of first sampling the
(empirical) CDF of the p-values at each node and then forwarding
$\mathcal{O}(\log m)$ bits to its neighbors. Under the same assumptions as for
the original BH procedure, our method has both the provable finite-sample FDR
control as well as competitive empirical detection power, even with a few
samples at each node. We provide an asymptotic analysis of power under a
mixture model assumption on the p-values.
( 2
min )
We characterise the learning of a mixture of two clouds of data points with
generic centroids via empirical risk minimisation in the high dimensional
regime, under the assumptions of generic convex loss and convex regularisation.
Each cloud of data points is obtained by sampling from a possibly uncountable
superposition of Gaussian distributions, whose variance has a generic
probability density $\varrho$. Our analysis covers therefore a large family of
data distributions, including the case of power-law-tailed distributions with
no covariance. We study the generalisation performance of the obtained
estimator, we analyse the role of regularisation, and the dependence of the
separability transition on the distribution scale parameters.
( 2
min )
D-Adaptation is an approach to automatically setting the learning rate which
asymptotically achieves the optimal rate of convergence for minimizing convex
Lipschitz functions, with no back-tracking or line searches, and no additional
function value or gradient evaluations per step. Our approach is the first
hyper-parameter free method for this class without additional multiplicative
log factors in the convergence rate. We present extensive experiments for SGD
and Adam variants of our method, where the method automatically matches
hand-tuned learning rates across more than a dozen diverse machine learning
problems, including large-scale vision and language problems.
An open-source implementation is available.
( 2
min )
This study explores the number of neurons required for a Rectified Linear
Unit (ReLU) neural network to approximate multivariate monomials. We establish
an exponential lower bound on the complexity of any shallow network
approximating the product function over a general compact domain. We also
demonstrate this lower bound doesn't apply to normalized Lipschitz monomials
over the unit cube. These findings suggest that shallow ReLU networks
experience the curse of dimensionality when expressing functions with a
Lipschitz parameter scaling with the dimension of the input, and that the
expressive power of neural networks is more dependent on their depth rather
than overall complexity.
( 2
min )
We present a novel framework for distributionally robust optimization (DRO),
called cost-aware DRO (CADRO). The key idea of CADRO is to exploit the cost
structure in the design of the ambiguity set to reduce conservatism.
Particularly, the set specifically constrains the worst-case distribution along
the direction in which the expected cost of an approximate solution increases
most rapidly. We prove that CADRO provides both a high-confidence upper bound
and a consistent estimator of the out-of-sample expected cost, and show
empirically that it produces solutions that are substantially less conservative
than existing DRO methods, while providing the same guarantees.
( 2
min )
Denoising diffusion models are a class of generative models which have
recently achieved state-of-the-art results across many domains. Gradual noise
is added to the data using a diffusion process, which transforms the data
distribution into a Gaussian. Samples from the generative model are then
obtained by simulating an approximation of the time reversal of this diffusion
initialized by Gaussian samples. Recent research has explored adapting
diffusion models for sampling and inference tasks. In this paper, we leverage
known connections to stochastic control akin to the F\"ollmer drift to extend
established neural network approximation results for the F\"ollmer drift to
denoising diffusion models and samplers.
( 2
min )
A common pipeline in learning-based control is to iteratively estimate a
model of system dynamics, and apply a trajectory optimization algorithm -
e.g.~$\mathtt{iLQR}$ - on the learned model to minimize a target cost. This
paper conducts a rigorous analysis of a simplified variant of this strategy for
general nonlinear systems. We analyze an algorithm which iterates between
estimating local linear models of nonlinear system dynamics and performing
$\mathtt{iLQR}$-like policy updates. We demonstrate that this algorithm attains
sample complexity polynomial in relevant problem parameters, and, by
synthesizing locally stabilizing gains, overcomes exponential dependence in
problem horizon. Experimental results validate the performance of our
algorithm, and compare to natural deep-learning baselines.
( 2
min )
While Convolutional Neural Networks (CNNs) have long been investigated and
applied, as well as theorized, we aim to provide a slightly different
perspective into their nature -- through the perspective of their Hessian maps.
The reason is that the loss Hessian captures the pairwise interaction of
parameters and therefore forms a natural ground to probe how the architectural
aspects of CNN get manifested in its structure and properties. We develop a
framework relying on Toeplitz representation of CNNs, and then utilize it to
reveal the Hessian structure and, in particular, its rank. We prove tight upper
bounds (with linear activations), which closely follow the empirical trend of
the Hessian rank and hold in practice in more general settings. Overall, our
work generalizes and establishes the key insight that, even in CNNs, the
Hessian rank grows as the square root of the number of parameters.
( 2
min )
ChatGPT is a sophisticated language model that has taken the world by storm. With its advanced natural language processing capabilities and…
( 11
min )
Some reactions to the latest AI news and developments, along with some AI-generated artwork. Follow the channel, to get updates on posts.
( 13
min )
Leo Anthony Celi invites industry to broaden its focus in gathering and analyzing clinical data for every population.
( 9
min )
Announcements LLM success depends on quality, transparent data Everyone from writers to coders wonder if their job is in jeopardy as prognosticators say generative AI tools will take over business in the coming years. Of course, these large language model chatbots are still unreliable, and certainly can’t be trusted to complete jobs as well as… Read More »DSC Weekly 16 May 2023 – LLM success depends on quality, transparent data
The post DSC Weekly 16 May 2023 – LLM success depends on quality, transparent data appeared first on Data Science Central.
( 19
min )
Image sourced from striim.com There’s one universal truth for every modern organization. It doesn’t matter whether you’re starting a business or already established: to succeed, you need data. Of course, not just any data will do. For strong data-driven decision-making, you also need the best insights. Thankfully, due to data analytics tools, businesses of all… Read More »6 Reasons Real-Time Data Analytics is Beneficial for Your Business
The post 6 Reasons Real-Time Data Analytics is Beneficial for Your Business appeared first on Data Science Central.
( 24
min )
In today’s technological world, data is everything. It can inform our marketing decisions, improve product creation, boost internal processes, and more. For an online business, having the best possible data is key to success. But simply having data isn’t enough. To obtain useful information, you need to understand your data. That’s where web data analytics… Read More »5 Ways to Use Analytics to Inform Website Development Decisions
The post 5 Ways to Use Analytics to Inform Website Development Decisions appeared first on Data Science Central.
( 24
min )
During the past decade, the publishing industry has undergone significant transformations due to the development of digital platforms and the widespread availability of user-generated content. Although these advancements have enabled a greater availability of information and a more diverse perspective, they have also presented challenges when it comes to ensuring that the content adheres to… Read More »Publishing Industry: The Extreme Crucial Role of AI in Content Moderation
The post Publishing Industry: The Extreme Crucial Role of AI in Content Moderation appeared first on Data Science Central.
( 22
min )
Today we are excited to announce that Together Computer’s GPT-NeoXT-Chat-Base-20B language foundation model is available for customers using Amazon SageMaker JumpStart. GPT-NeoXT-Chat-Base-20B is an open-source model to build conversational bots. You can easily try out this model and use it with JumpStart. JumpStart is the machine learning (ML) hub of Amazon SageMaker that provides access […]
( 12
min )
This research was accepted by the IEEE/ACM International Conference on Software Engineering (ICSE), which is a forum for researchers, practitioners, and educators to gather, present, and discuss the most recent innovations, trends, experiences, and issues in the field of software engineering. The Microsoft 365 Systems Innovation research group has a paper accepted at the 45th […]
The post Large-language models for automatic cloud incident management appeared first on Microsoft Research.
( 11
min )
Ten thousand years after the last woolly mammoths vanished with the last Ice Age, a team of computational biologists is on a mission to bring them back within five years. Led by synthetic biology pioneer George Church, Colossal Biosciences is also seeking to return the dodo bird and Tasmanian tiger, as well as help save Read article >
( 7
min )
Chip manufacturing is an “ideal application” for NVIDIA accelerated and AI computing, NVIDIA founder and CEO Jensen Huang said Tuesday. Detailing how the latest advancements in computing are accelerating “the world’s most important industry,” Huang spoke at ITF World 2023 semiconductor conference in Antwerp, Belgium. Huang delivered his remarks via video to a gathering of Read article >
( 7
min )
Data Science is a popular as well as vast field; till date, there are a lot of opportunities in this field, and most people, whether they…
( 25
min )
Introduction
( 17
min )
This is a guest post co-authored by Nafi Ahmet Turgut, Mutlu Polatcan, Pınar Baki, Mehmet İkbal Özmen, Hasan Burak Yel, and Hamza Akyıldız from Getir. Getir is the pioneer of ultrafast grocery delivery. The tech company has revolutionized last-mile delivery with its “groceries in minutes” delivery proposition. Getir was founded in 2015 and operates in […]
( 8
min )
Amazon Textract is a machine learning (ML) service that automatically extracts text, handwriting, and data from any document or image. To make it simpler to evaluate the capabilities of Amazon Textract, we have launched a new Bulk Document Uploader feature on the Amazon Textract console that enables you to quickly process your own set of […]
( 7
min )
$\partial\mathbb{B}$ nets are differentiable neural networks that learn
discrete boolean-valued functions by gradient descent. $\partial\mathbb{B}$
nets have two semantically equivalent aspects: a differentiable soft-net, with
real weights, and a non-differentiable hard-net, with boolean weights. We train
the soft-net by backpropagation and then `harden' the learned weights to yield
boolean weights that bind with the hard-net. The result is a learned discrete
function. `Hardening' involves no loss of accuracy, unlike existing approaches
to neural network binarization. Preliminary experiments demonstrate that
$\partial\mathbb{B}$ nets achieve comparable performance on standard machine
learning problems yet are compact (due to 1-bit weights) and interpretable (due
to the logical nature of the learnt functions).
( 2
min )
Understanding material surfaces and interfaces is vital in applications like
catalysis or electronics. Ab initio simulations, combining energies from
electronic structure with statistical mechanics, can, in principle, predict the
structure of material surfaces as a function of thermodynamic variables.
However, accurate energy simulations are prohibitive when coupled to the vast
phase space that must be statistically sampled. Here, we present a bi-faceted
computational loop to predict surface phase diagrams of multi-component
materials that accelerates both the energy scoring and statistical sampling
methods. Fast, scalable, and data-efficient machine learning interatomic
potentials are trained on high-throughput density-functional theory
calculations through closed-loop active learning. Markov-chain Monte Carlo
sampling in the semi-grand canonical ensemble is enabled by using virtual
surface sites. The predicted surfaces for GaN(0001) and SrTiO3(001) are in
agreement with past work and suggest that the proposed strategy can model
complex material surfaces and discover previously unreported surface
terminations.
( 2
min )
Pre-trained language models (PLMs) are known to improve the generalization
performance of natural language understanding models by leveraging large
amounts of data during the pre-training phase. However, the out-of-distribution
(OOD) generalization problem remains a challenge in many NLP tasks, limiting
the real-world deployment of these methods. This paper presents the first
attempt at creating a unified benchmark named \method for evaluating OOD
robustness in NLP models, highlighting the importance of OOD robustness and
providing insights on how to measure the robustness of a model and how to
improve it. The benchmark includes 13 publicly available datasets for OOD
testing, and evaluations are conducted on 8 classic NLP tasks over 21 popularly
used PLMs, including GPT-3 and GPT-3.5. Our findings confirm the need for
improved OOD accuracy in NLP tasks, as significant performance degradation was
observed in all settings compared to in-distribution (ID) accuracy.
( 2
min )
Given data ${\rm X}\in\mathbb{R}^{n\times d}$ and labels
$\mathbf{y}\in\mathbb{R}^{n}$ the goal is find $\mathbf{w}\in\mathbb{R}^d$ to
minimize $\Vert{\rm X}\mathbf{w}-\mathbf{y}\Vert^2$. We give a polynomial
algorithm that, \emph{oblivious to $\mathbf{y}$}, throws out $n/(d+\sqrt{n})$
data points and is a $(1+d/n)$-approximation to optimal in expectation. The
motivation is tight approximation with reduced label complexity (number of
labels revealed). We reduce label complexity by $\Omega(\sqrt{n})$. Open
question: Can label complexity be reduced by $\Omega(n)$ with tight
$(1+d/n)$-approximation?
( 2
min )
Breast cancer early detection is crucial for improving patient outcomes. The
Institut Catal\`a de la Salut (ICS) has launched the DigiPatICS project to
develop and implement artificial intelligence algorithms to assist with the
diagnosis of cancer. In this paper, we propose a new approach for facing the
color normalization problem in HER2-stained histopathological images of breast
cancer tissue, posed as an style transfer problem. We combine the Color
Deconvolution technique with the Pix2Pix GAN network to present a novel
approach to correct the color variations between different HER2 stain brands.
Our approach focuses on maintaining the HER2 score of the cells in the
transformed images, which is crucial for the HER2 analysis. Results demonstrate
that our final model outperforms the state-of-the-art image style transfer
methods in maintaining the cell classes in the transformed images and is as
effective as them in generating realistic images.
( 2
min )
Over the last few years, the number of works about deep learning applied to
the medical field has increased enormously. The necessity of a rigorous
assessment of these models is required to explain these results to all people
involved in medical exams. A recent field in the machine learning area is
explainable artificial intelligence, also known as XAI, which targets to
explain the results of such black box models to permit the desired assessment.
This survey analyses several recent studies in the XAI field applied to medical
diagnosis research, allowing some explainability of the machine learning
results in several different diseases, such as cancers and COVID-19.
( 2
min )
This paper proposes an online, provably robust, and scalable Bayesian
approach for changepoint detection. The resulting algorithm has key advantages
over previous work: it provides provable robustness by leveraging the
generalised Bayesian perspective, and also addresses the scalability issues of
previous attempts. Specifically, the proposed generalised Bayesian formalism
leads to conjugate posteriors whose parameters are available in closed form by
leveraging diffusion score matching. The resulting algorithm is exact, can be
updated through simple algebra, and is more than 10 times faster than its
closest competitor.
( 2
min )
In the misspecified kernel ridge regression problem, researchers usually
assume the underground true function $f_{\rho}^{*} \in [\mathcal{H}]^{s}$, a
less-smooth interpolation space of a reproducing kernel Hilbert space (RKHS)
$\mathcal{H}$ for some $s\in (0,1)$. The existing minimax optimal results
require $\|f_{\rho}^{*}\|_{L^{\infty}}
\alpha_{0}$ where $\alpha_{0}\in (0,1)$ is the embedding index, a constant
depending on $\mathcal{H}$. Whether the KRR is optimal for all $s\in (0,1)$ is
an outstanding problem lasting for years. In this paper, we show that KRR is
minimax optimal for any $s\in (0,1)$ when the $\mathcal{H}$ is a Sobolev RKHS.
( 2
min )
Anonymization techniques based on obfuscating the quasi-identifiers by means
of value generalization hierarchies are widely used to achieve preset levels of
privacy. To prevent different types of attacks against database privacy it is
necessary to apply several anonymization techniques beyond the classical
k-anonymity or $\ell$-diversity. However, the application of these methods is
directly connected to a reduction of their utility in prediction and decision
making tasks. In this work we study four classical machine learning methods
currently used for classification purposes in order to analyze the results as a
function of the anonymization techniques applied and the parameters selected
for each of them. The performance of these models is studied when varying the
value of k for k-anonymity and additional tools such as $\ell$-diversity,
t-closeness and $\delta$-disclosure privacy are also deployed on the well-known
adult dataset.
( 2
min )
The Schr\"odinger bridge problem (SBP) is gaining increasing attention in
generative modeling and showing promising potential even in comparison with the
score-based generative models (SGMs). SBP can be interpreted as an
entropy-regularized optimal transport problem, which conducts projections onto
every other marginal alternatingly. However, in practice, only approximated
projections are accessible and their convergence is not well understood. To
fill this gap, we present a first convergence analysis of the Schr\"odinger
bridge algorithm based on approximated projections. As for its practical
applications, we apply SBP to probabilistic time series imputation by
generating missing values conditioned on observed data. We show that optimizing
the transport cost improves the performance and the proposed algorithm achieves
the state-of-the-art result in healthcare and environmental data while
exhibiting the advantage of exploring both temporal and feature patterns in
probabilistic time series imputation.
( 2
min )
The use of deep learning approaches for image reconstruction is of
contemporary interest in radiology, especially for approaches that solve
inverse problems associated with imaging. In deployment, these models may be
exposed to input distributions that are widely shifted from training data, due
in part to data biases or drifts. We propose a metric based on local Lipschitz
determined from a single trained model that can be used to estimate the model
uncertainty for image reconstructions. We demonstrate a monotonic relationship
between the local Lipschitz value and Mean Absolute Error and show that this
method can be used to provide a threshold that determines whether a given DL
reconstruction approach was well suited to the task. Our uncertainty estimation
method can be used to identify out-of-distribution test samples, relate
information regarding epistemic uncertainties, and guide proper data
augmentation. Quantifying uncertainty of learned reconstruction approaches is
especially pertinent to the medical domain where reconstructed images must
remain diagnostically accurate.
( 2
min )
Recent works successfully leveraged Large Language Models' (LLM) abilities to
capture abstract knowledge about world's physics to solve decision-making
problems. Yet, the alignment between LLMs' knowledge and the environment can be
wrong and limit functional competence due to lack of grounding. In this paper,
we study an approach (named GLAM) to achieve this alignment through functional
grounding: we consider an agent using an LLM as a policy that is progressively
updated as the agent interacts with the environment, leveraging online
Reinforcement Learning to improve its performance to solve goals. Using an
interactive textual environment designed to study higher-level forms of
functional grounding, and a set of spatial and navigation tasks, we study
several scientific questions: 1) Can LLMs boost sample efficiency for online
learning of various RL tasks? 2) How can it boost different forms of
generalization? 3) What is the impact of online learning? We study these
questions by functionally grounding several variants (size, architecture) of
FLAN-T5.
( 2
min )
This paper discusses the feasibility of continuously training the CLIP model
through streaming data. Then, by tracking the directional changes of the
representation vectors in the continuously updated CLIP model, we explore and
summarize these spatial variations as Spatial Disorder (SD), which can be
divided into Intra-modal Rotation and Inter-modal Deviation. Moreover, we
demonstrate how intra-modal rotation and inter-modal deviation lead to a
performance decline for CLIP on cross-modal retrieval tasks in both empirically
and theoretically. To alleviate the spatial disorder, we propose a simple yet
effective continual learning framework Mod-X: Maintain off-diagonal
information-matriX. The experiments (in Section \ref{method}, \ref{experiments}
and Appendix \ref{Appendix_to_experiments}) on commonly used datasets with
different scales and scopes have illustrated the effectiveness of our method.
( 2
min )
Critical decisions like loan approvals, medical interventions, and college
admissions are guided by predictions made in the presence of uncertainty. In
this paper, we prove that uncertainty has a disparate impact. While it imparts
errors across all demographic groups, the types of errors vary systematically:
Groups with higher average outcomes are typically assigned higher false
positive rates, while those with lower average outcomes are assigned higher
false negative rates. We show that additional data acquisition can eliminate
the disparity and broaden access to opportunity. The strategy, which we call
Affirmative Information, could stand as an alternative to Affirmative Action.
( 2
min )
In this work, we focus on the communication aspect of decentralized learning,
which involves multiple agents training a shared machine learning model using
decentralized stochastic gradient descent (D-SGD) over distributed data. In
particular, we investigate the impact of broadcast transmission and
probabilistic random access policy on the convergence performance of D-SGD,
considering the broadcast nature of wireless channels and the link dynamics in
the communication topology. Our results demonstrate that optimizing the access
probability to maximize the expected number of successful links is a highly
effective strategy for accelerating the system convergence.
( 2
min )
Existing question answering methods often assume that the input content
(e.g., documents or videos) is always accessible to solve the task.
Alternatively, memory networks were introduced to mimic the human process of
incremental comprehension and compression of the information in a
fixed-capacity memory. However, these models only learn how to maintain memory
by backpropagating errors in the answers through the entire network. Instead,
it has been suggested that humans have effective mechanisms to boost their
memorization capacities, such as rehearsal and anticipation. Drawing
inspiration from these, we propose a memory model that performs rehearsal and
anticipation while processing inputs to memorize important information for
solving question answering tasks from streaming data. The proposed mechanisms
are applied self-supervised during training through masked modeling tasks
focused on coreference information. We validate our model on a short-sequence
(bAbI) dataset as well as large-sequence textual (NarrativeQA) and video
(ActivityNet-QA) question answering datasets, where it achieves substantial
improvements over previous memory network approaches. Furthermore, our ablation
study confirms the proposed mechanisms' importance for memory models.
( 2
min )
Diffusion models were initially developed for text-to-image generation and
are now being utilized to generate high quality synthetic images. Preceded by
GANs, diffusion models have shown impressive results using various evaluation
metrics. However, commonly used metrics such as FID and IS are not suitable for
determining whether diffusion models are simply reproducing the training
images. Here we train StyleGAN and diffusion models, using BRATS20 and BRATS21
datasets, to synthesize brain tumor images, and measure the correlation between
the synthetic images and all training images. Our results show that diffusion
models are much more likely to memorize the training images, especially for
small datasets. Researchers should be careful when using diffusion models for
medical imaging, if the final goal is to share the synthetic images.
( 2
min )
We present GPS++, a hybrid Message Passing Neural Network / Graph Transformer
model for molecular property prediction. Our model integrates a well-tuned
local message passing component and biased global attention with other key
ideas from prior literature to achieve state-of-the-art results on large-scale
molecular dataset PCQM4Mv2. Through a thorough ablation study we highlight the
impact of individual components and find that nearly all of the model's
performance can be maintained without any use of global self-attention, showing
that message passing is still a competitive approach for 3D molecular property
prediction despite the recent dominance of graph transformers. We also find
that our approach is significantly more accurate than prior art when 3D
positional information is not available.
( 2
min )
This field case study aims to address the challenge of accurately predicting
petrophysical properties in heterogeneous reservoir formations, which can
significantly impact reservoir performance predictions. The study employed
three machine learning algorithms, namely Artificial Neural Network (ANN),
Random Forest Classifier (RFC), and Support Vector Machine (SVM), to predict
permeability log from conventional logs and match it with core data. The
primary objective of this study was to compare the effectiveness of the three
machine learning algorithms in predicting permeability and determine the
optimal prediction method. The study utilized the Flow Zone Indicator (FZI)
rock typing technique to understand the factors influencing reservoir quality.
The findings will be used to improve reservoir simulation and locate future
wells more accurately. The study concluded that the FZI approach and machine
learning algorithms are effective in predicting permeability log and improving
reservoir performance predictions.
( 2
min )
Classical reinforcement learning (RL) aims to optimize the expected
cumulative reward. In this work, we consider the RL setting where the goal is
to optimize the quantile of the cumulative reward. We parameterize the policy
controlling actions by neural networks, and propose a novel policy gradient
algorithm called Quantile-Based Policy Optimization (QPO) and its variant
Quantile-Based Proximal Policy Optimization (QPPO) for solving deep RL problems
with quantile objectives. QPO uses two coupled iterations running at different
timescales for simultaneously updating quantiles and policy parameters, whereas
QPPO is an off-policy version of QPO that allows multiple updates of parameters
during one simulation episode, leading to improved algorithm efficiency. Our
numerical results indicate that the proposed algorithms outperform the existing
baseline algorithms under the quantile criterion.
( 2
min )
With the publication of DINO, a variant of the Detection Transformer (DETR),
Detection Transformers are breaking the record in the object detection
benchmark with the merits of their end-to-end design and scalability. However,
the extension of DETR to oriented object detection has not been thoroughly
studied although more benefits from its end-to-end architecture are expected
such as removing NMS and anchor-related costs. In this paper, we propose a
first strong DINO-based baseline for oriented object detection. We found that
straightforward employment of DETRs for oriented object detection does not
guarantee non-duplicate prediction, and propose a simple cost to mitigate this.
Furthermore, we introduce a novel denoising strategy that uses Hungarian
matching to filter redundant noised queries and query alignment to preserve
matching consistency between Transformer decoder layers. Our proposed model
outperforms previous rotated DETRs and other counterparts, achieving
state-of-the-art performance in DOTA-v1.0/v1.5/v2.0, and DIOR-R benchmarks.
( 2
min )
We propose the GFlowNets with Human Feedback (GFlowHF) framework to improve
the exploration ability when training AI models. For tasks where the reward is
unknown, we fit the reward function through human evaluations on different
trajectories. The goal of GFlowHF is to learn a policy that is strictly
proportional to human ratings, instead of only focusing on human favorite
ratings like RLHF. Experiments show that GFlowHF can achieve better exploration
ability than RLHF.
( 2
min )
We present the Hierarchical Mixture Networks (HINT), a model family for
efficient and accurate coherent forecasting. We specialize the networks on the
task via a multivariate mixture optimized with composite likelihood and made
coherent via bootstrap reconciliation. Additionally, we robustify the networks
to stark time series scale variations, incorporating normalized feature
extraction and recomposition of output scales within their architecture. We
demonstrate 8% sCRPS improved accuracy across five datasets compared to the
existing state-of-the-art. We conduct ablation studies on our model's
components and extensively investigate the theoretical properties of the
multivariate mixture. HINT's code is available at this
https://github.com/Nixtla/neuralforecast.
( 2
min )
This work proposes the use of 3D convolutional variational autoencoders
(CVAEs) to trace the changes and symptomatology produced by neurodegeneration
in Parkinson's disease (PD). In this work, we present a novel approach to
detect and quantify changes in dopamine transporter (DaT) concentration and its
spatial patterns using 3D CVAEs on Ioflupane (FPCIT) imaging. Our approach
leverages the power of deep learning to learn a low-dimensional representation
of the brain imaging data, which then is linked to different symptom categories
using regression algorithms. We demonstrate the effectiveness of our approach
on a dataset of PD patients and healthy controls, and show that general
symptomatology (UPDRS) is linked to a d-dimensional decomposition via the CVAE
with R2>0.25. Our work shows the potential of representation learning not only
in early diagnosis but in understanding neurodegeneration processes and
symptomatology.
( 2
min )
This paper proposes an online, provably robust, and scalable Bayesian
approach for changepoint detection. The resulting algorithm has key advantages
over previous work: it provides provable robustness by leveraging the
generalised Bayesian perspective, and also addresses the scalability issues of
previous attempts. Specifically, the proposed generalised Bayesian formalism
leads to conjugate posteriors whose parameters are available in closed form by
leveraging diffusion score matching. The resulting algorithm is exact, can be
updated through simple algebra, and is more than 10 times faster than its
closest competitor.
( 2
min )
We present the Hierarchical Mixture Networks (HINT), a model family for
efficient and accurate coherent forecasting. We specialize the networks on the
task via a multivariate mixture optimized with composite likelihood and made
coherent via bootstrap reconciliation. Additionally, we robustify the networks
to stark time series scale variations, incorporating normalized feature
extraction and recomposition of output scales within their architecture. We
demonstrate 8% sCRPS improved accuracy across five datasets compared to the
existing state-of-the-art. We conduct ablation studies on our model's
components and extensively investigate the theoretical properties of the
multivariate mixture. HINT's code is available at this
https://github.com/Nixtla/neuralforecast.
( 2
min )
Critical decisions like loan approvals, medical interventions, and college
admissions are guided by predictions made in the presence of uncertainty. In
this paper, we prove that uncertainty has a disparate impact. While it imparts
errors across all demographic groups, the types of errors vary systematically:
Groups with higher average outcomes are typically assigned higher false
positive rates, while those with lower average outcomes are assigned higher
false negative rates. We show that additional data acquisition can eliminate
the disparity and broaden access to opportunity. The strategy, which we call
Affirmative Information, could stand as an alternative to Affirmative Action.
( 2
min )
In Part 1 of the series “AI for Everyone: Learn How to Think Like a Data Scientist”, we discussed that for AI to reach its full economic and societal potential, we must educate and empower everyone to actively participate in the design, application, and management of meaningful, relevant, and responsible AI. We discussed the role… Read More »AI for Everyone: Learn How to Think Like a Data Scientist – Part 2
The post AI for Everyone: Learn How to Think Like a Data Scientist – Part 2 appeared first on Data Science Central.
( 19
min )
Deep machine learning models including Convolutional Neural Networks (CNN)
have been successful in the detection of Mild Cognitive Impairment (MCI) using
medical images, questionnaires, and videos. This paper proposes a novel
Multi-branch Classifier-Video Vision Transformer (MC-ViViT) model to
distinguish MCI from those with normal cognition by analyzing facial features.
The data comes from the I-CONECT, a behavioral intervention trial aimed at
improving cognitive function by providing frequent video chats. MC-ViViT
extracts spatiotemporal features of videos in one branch and augments
representations by the MC module. The I-CONECT dataset is challenging as the
dataset is imbalanced containing Hard-Easy and Positive-Negative samples, which
impedes the performance of MC-ViViT. We propose a loss function for Hard-Easy
and Positive-Negative Samples (HP Loss) by combining Focal loss and AD-CORRE
loss to address the imbalanced problem. Our experimental results on the
I-CONECT dataset show the great potential of MC-ViViT in predicting MCI with a
high accuracy of 90.63\% accuracy on some of the interview videos.
( 2
min )
The use of Artificial Intelligence (AI) has become increasingly prevalent in the modern world, seeing its potential to drastically improve…
( 25
min )
Amazon SageMaker comes with two options to spin up fully managed notebooks for exploring data and building machine learning (ML) models. The first option is fast start, collaborative notebooks accessible within Amazon SageMaker Studio—a fully integrated development environment (IDE) for machine learning. You can quickly launch notebooks in Studio, easily dial up or down the […]
( 9
min )
Amazon Kendra is an intelligent search service powered by machine learning (ML). Amazon Kendra reimagines enterprise search for your websites and applications so your employees and customers can easily find the content they’re looking for, even when it’s scattered across multiple locations and content repositories within your organization. Keywords or natural language questions can be […]
( 7
min )
EuroSys 2023 is the premier systems conference in Europe, and 2023 marks its 18th edition. Sponsored by ACM SIGOPS Europe and hosted May 8 to May 12, the conference covers a wide range of topics, including operating systems, real-time and networked systems, storage and middleware, and distributed, parallel, and embedded computing, as well as their […]
The post Microsoft at EuroSys 2023: Systems innovation across the stack to help support an easier, faster, safer, and smarter cloud appeared first on Microsoft Research.
( 11
min )
We consider the upper confidence bound strategy for Gaussian multi-armed
bandits with known control horizon sizes $N$ and build its limiting description
with a system of stochastic differential equations and ordinary differential
equations. Rewards for the arms are assumed to have unknown expected values and
known variances. A set of Monte-Carlo simulations was performed for the case of
close distributions of rewards, when mean rewards differ by the magnitude of
order $N^{-1/2}$, as it yields the highest normalized regret, to verify the
validity of the obtained description. The minimal size of the control horizon
when the normalized regret is not noticeably larger than maximum possible was
estimated.
( 2
min )
Due to the huge amount of parameters, fine-tuning of pretrained language
models (PLMs) is prone to overfitting in the low resource scenarios. In this
work, we present a novel method that operates on the hidden representations of
a PLM to reduce overfitting. During fine-tuning, our method inserts random
autoencoders between the hidden layers of a PLM, which transform activations
from the previous layers into multi-view compressed representations before
feeding them into the upper layers. The autoencoders are plugged out after
fine-tuning, so our method does not add extra parameters or increase
computation cost during inference. Our method demonstrates promising
performance improvement across a wide range of sequence- and token-level
low-resource NLP tasks.
( 2
min )
The Nash Equilibrium (NE) estimation in bidding games of electricity markets
is the key concern of both generation companies (GENCOs) for bidding strategy
optimization and the Independent System Operator (ISO) for market surveillance.
However, existing methods for NE estimation in emerging modern electricity
markets (FEM) are inaccurate and inefficient because the priori knowledge of
bidding strategies before any environment changes, such as load demand
variations, network congestion, and modifications of market design, is not
fully utilized. In this paper, a Bayes-adaptive Markov Decision Process in FEM
(BAMDP-FEM) is therefore developed to model the GENCOs' bidding strategy
optimization considering the priori knowledge. A novel Multi-Agent Generative
Adversarial Imitation Learning algorithm (MAGAIL-FEM) is then proposed to
enable GENCOs to learn simultaneously from priori knowledge and interactions
with changing environments. The obtained NE is a Bayesian Nash Equilibrium
(BNE) with priori knowledge transferred from the previous environment. In the
case study, the superiority of this proposed algorithm in terms of convergence
speed compared with conventional methods is verified. It is concluded that the
optimal bidding strategies in the obtained BNE can always lead to more profits
than NE due to the effective learning from the priori knowledge. Also, BNE is
more accurate and consistent with situations in real-world markets.
( 2
min )
Implicitly Normalized Forecaster (online mirror descent with Tsallis entropy
as prox-function) is known to be an optimal algorithm for adversarial
multi-armed problems (MAB). However, most of the complexity results rely on
bounded rewards or other restrictive assumptions. Recently closely related
best-of-both-worlds algorithm were proposed for both adversarial and stochastic
heavy-tailed MAB settings. This algorithm is known to be optimal in both
settings, but fails to exploit data fully. In this paper, we propose Implicitly
Normalized Forecaster with clipping for MAB problems with heavy-tailed
distribution on rewards. We derive convergence results under mild assumptions
on rewards distribution and show that the proposed method is optimal for both
linear and non-linear heavy-tailed stochastic MAB problems. Also we show that
algorithm usually performs better compared to best-of-two-worlds algorithm.
( 2
min )
Neural ordinary differential equations (neural ODEs) are a popular family of
continuous-depth deep learning models. In this work, we consider a large family
of parameterized ODEs with continuous-in-time parameters, which include
time-dependent neural ODEs. We derive a generalization bound for this class by
a Lipschitz-based argument. By leveraging the analogy between neural ODEs and
deep residual networks, our approach yields in particular a generalization
bound for a class of deep residual networks. The bound involves the magnitude
of the difference between successive weight matrices. We illustrate numerically
how this quantity affects the generalization capability of neural networks.
( 2
min )
Hyperparameter optimization (HPO) is a powerful technique for automating the
tuning of machine learning (ML) models. However, in many real-world
applications, accuracy is only one of multiple performance criteria that must
be considered. Optimizing these objectives simultaneously on a complex and
diverse search space remains a challenging task. In this paper, we propose
MO-DEHB, an effective and flexible multi-objective (MO) optimizer that extends
the recent evolutionary Hyperband method DEHB. We validate the performance of
MO-DEHB using a comprehensive suite of 15 benchmarks consisting of diverse and
challenging MO problems, including HPO, neural architecture search (NAS), and
joint NAS and HPO, with objectives including accuracy, latency and algorithmic
fairness. A comparative study against state-of-the-art MO optimizers
demonstrates that MO-DEHB clearly achieves the best performance across our 15
benchmarks.
( 2
min )
We propose a node clustering method for time-varying graphs based on the
assumption that the cluster labels are changed smoothly over time. Clustering
is one of the fundamental tasks in many science and engineering fields
including signal processing, machine learning, and data mining. Although most
existing studies focus on the clustering of nodes in static graphs, we often
encounter time-varying graphs for time-series data, e.g., social networks,
brain functional connectivity, and point clouds. In this paper, we formulate a
node clustering of time-varying graphs as an optimization problem based on
spectral clustering, with a smoothness constraint of the node labels. We solve
the problem with a primal-dual splitting algorithm. Experiments on synthetic
and real-world time-varying graphs are performed to validate the effectiveness
of the proposed approach.
( 2
min )
As a consequence of the increasing influence of machine learning on our
lives, everyone needs competencies to understand corresponding phenomena, but
also to get involved in shaping our world and making informed decisions
regarding the influences on our society. Therefore, in K-12 education, students
need to learn about core ideas and principles of machine learning. However, for
this target group, achieving all of the aforementioned goals presents an
enormous challenge. To this end, we present a teaching concept that combines a
playful and accessible unplugged approach focusing on conceptual understanding
with empowering students to actively apply machine learning methods and reflect
their influence on society, building upon decision tree learning.
( 2
min )
We study multi-agent reinforcement learning in the setting of episodic Markov
decision processes, where multiple agents cooperate via communication through a
central server. We propose a provably efficient algorithm based on value
iteration that enable asynchronous communication while ensuring the advantage
of cooperation with low communication overhead. With linear function
approximation, we prove that our algorithm enjoys an
$\tilde{\mathcal{O}}(d^{3/2}H^2\sqrt{K})$ regret with
$\tilde{\mathcal{O}}(dHM^2)$ communication complexity, where $d$ is the feature
dimension, $H$ is the horizon length, $M$ is the total number of agents, and
$K$ is the total number of episodes. We also provide a lower bound showing that
a minimal $\Omega(dM)$ communication complexity is required to improve the
performance through collaboration.
( 2
min )
As a medical doctor in Nigeria, Tobi Olatunji knows the stress of practicing in Africa’s busy hospitals. As a machine-learning scientist, he has a prescription for it. “I worked at one of West Africa’s largest hospitals, where I would routinely see more than 30 patients a day — it’s a very hard job,” said Olatunji. Read article >
( 6
min )
Make gaming a priority this GFN Thursday — time’s running out to upgrade to a GeForce NOW Priority six-month membership at 40% off the normal price. Find out how new Priority members are using the cloud to get their game on. Plus, the week brings updates for some of the hottest games in the GeForce Read article >
( 5
min )
For telcos around the world, one of the biggest challenges to upgrading networks has always been the question, “If you build it, will they come?” Asia’s leading telco, Singtel, believes the key to helping customers innovate with AI across industries — for everything from traffic and video analytics to conversational AI avatars powered by large Read article >
( 5
min )
As previous representations for reinforcement learning cannot effectively
incorporate a human-intuitive understanding of the 3D environment, they usually
suffer from sub-optimal performances. In this paper, we present Semantic-aware
Neural Radiance Fields for Reinforcement Learning (SNeRL), which jointly
optimizes semantic-aware neural radiance fields (NeRF) with a convolutional
encoder to learn 3D-aware neural implicit representation from multi-view
images. We introduce 3D semantic and distilled feature fields in parallel to
the RGB radiance fields in NeRF to learn semantic and object-centric
representation for reinforcement learning. SNeRL outperforms not only previous
pixel-based representations but also recent 3D-aware representations both in
model-free and model-based reinforcement learning.
( 2
min )
Correlation matrix visualization is essential for understanding the
relationships between variables in a dataset, but missing data can pose a
significant challenge in estimating correlation coefficients. In this paper, we
compare the effects of various missing data methods on the correlation plot,
focusing on two common missing patterns: random and monotone. We aim to provide
practical strategies and recommendations for researchers and practitioners in
creating and analyzing the correlation plot. Our experimental results suggest
that while imputation is commonly used for missing data, using imputed data for
plotting the correlation matrix may lead to a significantly misleading
inference of the relation between the features. We recommend using DPER, a
direct parameter estimation approach, for plotting the correlation matrix based
on its performance in the experiments.
( 2
min )
This paper focuses on solving a fault detection problem using multivariate
time series of vibration signals collected from planetary gearboxes in a test
rig. Various traditional machine learning and deep learning methods have been
proposed for multivariate time-series classification, including distance-based,
functional data-oriented, feature-driven, and convolution kernel-based methods.
Recent studies have shown using convolution kernel-based methods like ROCKET,
and 1D convolutional neural networks with ResNet and FCN, have robust
performance for multivariate time-series data classification. We propose an
ensemble of three convolution kernel-based methods and show its efficacy on
this fault detection problem by outperforming other approaches and achieving an
accuracy of more than 98.8\%.
( 2
min )
Likelihood-free inference methods typically make use of a distance between
simulated and real data. A common example is the maximum mean discrepancy
(MMD), which has previously been used for approximate Bayesian computation,
minimum distance estimation, generalised Bayesian inference, and within the
nonparametric learning framework. The MMD is commonly estimated at a root-$m$
rate, where $m$ is the number of simulated samples. This can lead to
significant computational challenges since a large $m$ is required to obtain an
accurate estimate, which is crucial for parameter estimation. In this paper, we
propose a novel estimator for the MMD with significantly improved sample
complexity. The estimator is particularly well suited for computationally
expensive smooth simulators with low- to mid-dimensional inputs. This claim is
supported through both theoretical results and an extensive simulation study on
benchmark simulators.
( 2
min )
In recent years neural networks have achieved impressive results on many
technological and scientific tasks. Yet, the mechanism through which these
models automatically select features, or patterns in data, for prediction
remains unclear. Identifying such a mechanism is key to advancing performance
and interpretability of neural networks and promoting reliable adoption of
these models in scientific applications. In this paper, we identify and
characterize the mechanism through which deep fully connected neural networks
learn features. We posit the Deep Neural Feature Ansatz, which states that
neural feature learning occurs by implementing the average gradient outer
product to up-weight features strongly related to model output. Our ansatz
sheds light on various deep learning phenomena including emergence of spurious
features and simplicity biases and how pruning networks can increase
performance, the "lottery ticket hypothesis." Moreover, the mechanism
identified in our work leads to a backpropagation-free method for feature
learning with any machine learning model. To demonstrate the effectiveness of
this feature learning mechanism, we use it to enable feature learning in
classical, non-feature learning models known as kernel machines and show that
the resulting models, which we refer to as Recursive Feature Machines, achieve
state-of-the-art performance on tabular data.
( 3
min )
In this paper we propose a new iterative algorithm to solve the fair PCA
(FPCA) problem. We start with the max-min fair PCA formulation originally
proposed in [1] and derive a simple and efficient iterative algorithm which is
based on the minorization-maximization (MM) approach. The proposed algorithm
relies on the relaxation of a semi-orthogonality constraint which is proved to
be tight at every iteration of the algorithm. The vanilla version of the
proposed algorithm requires solving a semi-definite program (SDP) at every
iteration, which can be further simplified to a quadratic program by
formulating the dual of the surrogate maximization problem. We also propose two
important reformulations of the fair PCA problem: a) fair robust PCA -- which
can handle outliers in the data, and b) fair sparse PCA -- which can enforce
sparsity on the estimated fair principal components. The proposed algorithms
are computationally efficient and monotonically increase their respective
design objectives at every iteration. An added feature of the proposed
algorithms is that they do not require the selection of any hyperparameter
(except for the fair sparse PCA case where a penalty parameter that controls
the sparsity has to be chosen by the user). We numerically compare the
performance of the proposed methods with two of the state-of-the-art approaches
on synthetic data sets and a real-life data set.
( 2
min )
We show the convergence of Wasserstein inverse reinforcement learning (WIRL)
for multi-objective optimizations with the projective subgradient method by
formulating an inverse problem of the optimization problem that is equivalent
to WIRL for multi-objective optimizations.
In addition, we prove convergence of inverse reinforcement learning (maximum
entropy inverse reinforcement learning, guid cost learning) for multi-objective
optimization with the projective subgradient method.
( 2
min )
We formulate a uniform tail bound for empirical processes indexed by a class
of functions, in terms of the individual deviations of the functions rather
than the worst-case deviation in the considered class. The tail bound is
established by introducing an initial "deflation" step to the standard generic
chaining argument. The resulting tail bound has a main complexity component, a
variant of Talagrand's $\gamma$ functional for the deflated function class, as
well as an instance-dependent deviation term, measured by an appropriately
scaled version of a suitable norm. Both of these terms are expressed using
certain coefficients formulated based on the relevant cumulant generating
functions. We also provide more explicit approximations for the mentioned
coefficients, when the function class lies in a given (exponential type) Orlicz
space.
( 2
min )
Robustness against image perturbations bounded by a $\ell_p$ ball have been
well-studied in recent literature. Perturbations in the real-world, however,
rarely exhibit the pixel independence that $\ell_p$ threat models assume. A
recently proposed Wasserstein distance-bounded threat model is a promising
alternative that limits the perturbation to pixel mass movements. We point out
and rectify flaws in previous definition of the Wasserstein threat model and
explore stronger attacks and defenses under our better-defined framework.
Lastly, we discuss the inability of current Wasserstein-robust models in
defending against perturbations seen in the real world. Our code and trained
models are available at https://github.com/edwardjhu/improved_wasserstein .
( 2
min )
While machine learning is currently transforming the field of histopathology,
the domain lacks a comprehensive evaluation of state-of-the-art models based on
essential but complementary quality requirements beyond a mere classification
accuracy. In order to fill this gap, we developed a new methodology to
extensively evaluate a wide range of classification models, including recent
vision transformers, and convolutional neural networks such as: ConvNeXt,
ResNet (BiT), Inception, ViT and Swin transformer, with and without supervised
or self-supervised pretraining. We thoroughly tested the models on five widely
used histopathology datasets containing whole slide images of breast, gastric,
and colorectal cancer and developed a novel approach using an image-to-image
translation model to assess the robustness of a cancer classification model
against stain variations. Further, we extended existing interpretability
methods to previously unstudied models and systematically reveal insights of
the models' classifications strategies that can be transferred to future model
architectures.
( 2
min )
This paper carries out sparse-penalized deep neural networks predictors for
learning weakly dependent processes, with a broad class of loss functions. We
deal with a general framework that includes, regression estimation,
classification, times series prediction, $\cdots$ The $\psi$-weak dependence
structure is considered, and for the specific case of bounded observations,
$\theta_\infty$-coefficients are also used. In this case of
$\theta_\infty$-weakly dependent, a non asymptotic generalization bound within
the class of deep neural networks predictors is provided. For learning both
$\psi$ and $\theta_\infty$-weakly dependent processes, oracle inequalities for
the excess risk of the sparse-penalized deep neural networks estimators are
established. When the target function is sufficiently smooth, the convergence
rate of these excess risk is close to $\mathcal{O}(n^{-1/3})$. Some simulation
results are provided, and application to the forecast of the particulate matter
in the Vit\'{o}ria metropolitan area is also considered.
( 2
min )
Creative studio Elara Systems doesn’t shy away from sensitive subjects in its work.
( 6
min )
Artificial intelligence is teaming with crowdsourcing to improve mRNA vaccines’ thermostability — the ability to avoid breaking down under heat stress — making distribution more accessible worldwide. In this episode of the NVIDIA AI Podcast, host Noah Kravitz interviews Bojan Tunguz, a physicist and senior system software engineer, and Johnny Israeli, senior manager of AI Read article >
( 5
min )
You can now register machine learning (ML) models built in Amazon SageMaker Canvas with a single click to the Amazon SageMaker Model Registry, enabling you to operationalize ML models in production. Canvas is a visual interface that enables business analysts to generate accurate ML predictions on their own—without requiring any ML experience or having to […]
( 8
min )
Today, data scientists who are training deep learning models need to identify and remediate model training issues to meet accuracy targets for production deployment, and require a way to utilize standard tools for debugging model training. Among the data scientist community, TensorBoard is a popular toolkit that allows data scientists to visualize and analyze various […]
( 8
min )
Amazon SageMaker provides a broad selection of machine learning (ML) infrastructure and model deployment options to help meet your ML inference needs. It’s a fully-managed service and integrates with MLOps tools so you can work to scale your model deployment, reduce inference costs, manage models more effectively in production, and reduce operational burden. SageMaker provides […]
( 7
min )
This is a guest post co-written with Trey Robinson, CTO at Sleepme Inc. Sleepme is an industry leader in sleep temperature management and monitoring products, including an Internet of Things (IoT) enabled sleep tracking sensor suite equipped with heart rate, respiration rate, bed and ambient temperature, humidity, and pressure sensors. Sleepme offers a smart mattress […]
( 6
min )
Understanding business trends, customer behavior, sales revenue, increase in demand, and buyer propensity all start with data. Exploring, analyzing, interpreting, and finding trends in data is essential for businesses to achieve successful outcomes. Business analysts play a pivotal role in facilitating data-driven business decisions through activities such as the visualization of business metrics and the […]
( 10
min )
Project Jupyter is a multi-stakeholder, open-source project that builds applications, open standards, and tools for data science, machine learning (ML), and computational science. The Jupyter Notebook, first released in 2011, has become a de facto standard tool used by millions of users worldwide across every possible academic, research, and industry sector. Jupyter enables users to […]
( 8
min )
Jupyter notebooks are highly favored by data scientists for their ability to interactively process data, build ML models, and test these models by making inferences on data. However, there are scenarios in which data scientists may prefer to transition from interactive development on notebooks to batch jobs. Examples of such use cases include scaling up […]
( 9
min )
Citadel founder and CEO Ken Griffin visits MIT, discusses how technology will continue to transform trading and investing.
( 9
min )
Models trained using common data-collection techniques judge rule violations more harshly than humans would, researchers report.
( 9
min )
Matt Shoulders will lead an interdisciplinary team to improve RuBisCO — the photosynthesis enzyme thought to be the holy grail for improving agricultural yield.
( 11
min )
A new computer vision system turns any shiny object into a camera of sorts, enabling an observer to see around corners or beyond obstructions.
( 10
min )
Learn 5 tips to implement cybersecurity awareness at your business and discover solutions to protect your business from cyber threats. Read on for more.
The post Building a Secure Workplace: 5 Strategies to Raise Cybersecurity Awareness appeared first on Data Science Central.
( 21
min )
Machine learning has emerged as a powerful tool for organizations across industries to enhance their operational efficiency and make data-driven decisions. With the increasing reliance of businesses on machine learning models, it is crucial to guarantee their performance as expected. At this point, monitoring the machine learning models comes into play. To put it simply,… Read More »You should never neglect to monitor your machine-learning models
The post You should never neglect to monitor your machine-learning models appeared first on Data Science Central.
( 21
min )
Amazon SageMaker Serverless Inference allows you to serve model inference requests in real time without having to explicitly provision compute instances or configure scaling policies to handle traffic variations. You can let AWS handle the undifferentiated heavy lifting of managing the underlying infrastructure and save costs in the process. A Serverless Inference endpoint spins up […]
( 13
min )
Proteins drive many biological processes, such as enzyme activity, molecular transport, and cellular support. The three-dimensional structure of a protein provides insight into its function and how it interacts with other biomolecules. Experimental methods to determine protein structure, such as X-ray crystallography and NMR spectroscopy, are expensive and time-consuming. In contrast, recently-developed computational methods can […]
( 8
min )
Healthcare data is complex and siloed, and exists in various formats. An estimated 80% of data within organizations is considered to be unstructured or “dark” data that is locked inside text, emails, PDFs, and scanned documents. This data is difficult to interpret or analyze programmatically and limits how organizations can derive insights from it and […]
( 7
min )
Amazon SageMaker provides a number of options for users who are looking for a solution to host their machine learning (ML) models. Of these options, one of the key features that SageMaker provides is real-time inference. Real-time inference workloads can have varying levels of requirements and service level agreements (SLAs) in terms of latency and […]
( 15
min )
3D artist Milan Dey finds inspiration in games, movies, comics and pop culture. He drew from all of the above when creating a stunning 3D scene of Mayan ruins, The Hidden Temple of Itzamná, this week In the NVIDIA Studio.
( 7
min )
Researchers identify a property that helps computer vision models learn to represent the visual world in a more stable, predictable way.
( 10
min )
With Amazon SageMaker, you can manage the whole end-to-end machine learning (ML) lifecycle. It offers many native capabilities to help manage ML workflows aspects, such as experiment tracking, and model governance via the model registry. This post provides a solution tailored to customers that are already using MLflow, an open-source platform for managing ML workflows. […]
( 15
min )
Sometimes it can be very beneficial to use tools such as compilers that can modify and compile your models for optimal inference performance. In this post, we explore TensorRT and how to use it with Amazon SageMaker inference using NVIDIA Triton Inference Server. We explore how TensorRT works and how to host and optimize these […]
( 15
min )
Warning: very long, 2-part blog series. But this topic is too important to not carefully explain how we can educate and empower everyone to participate in the AI conversation. Our success as a society depends upon our ability to include everyone in this conversation. “I love it when a plan comes together” – Hannibal Smith,… Read More »AI for Everyone: Learn How to Think Like a Data Scientist – Part 1
The post AI for Everyone: Learn How to Think Like a Data Scientist – Part 1 appeared first on Data Science Central.
( 22
min )
Machine learning in Healthcare industry.
The post How Machine Learning is Revolutionizing the Healthcare Industry appeared first on Data Science Central.
( 21
min )
Safety has been a critical issue for the deployment of learning-based
approaches in real-world applications. To address this issue, control barrier
function (CBF) and its variants have attracted extensive attention for
safety-critical control. However, due to the myopic one-step nature of CBF and
the lack of principled methods to design the class-$\mathcal{K}$ functions,
there are still fundamental limitations of current CBFs: optimality, stability,
and feasibility. In this paper, we proposed a novel and unified approach to
address these limitations with Adaptive Multi-step Control Barrier Function
(AM-CBF), where we parameterize the class-$\mathcal{K}$ function by a neural
network and train it together with the reinforcement learning policy. Moreover,
to mitigate the myopic nature, we propose a novel \textit{multi-step training
and single-step execution} paradigm to make CBF farsighted while the execution
remains solving a single-step convex quadratic program. Our method is evaluated
on the first and second-order systems in various scenarios, where our approach
outperforms the conventional CBF both qualitatively and quantitatively.
( 2
min )
Small molecules in biological samples are studied to provide information
about disease states, environmental toxins, natural product drug discovery, and
many other applications. The primary window into the composition of small
molecule mixtures is tandem mass spectrometry (MS2), which produces data that
are of high sensitivity and part per million resolution. We adopt multi-scale
sinusoidal embeddings of the mass data in MS2 designed to meet the challenge of
learning from the full resolution of MS2 data. Using these embeddings, we
provide a new state of the art model for spectral library search, the standard
task for initial evaluation of MS2 data. We also introduce a new task, chemical
property prediction from MS2 data, that has natural applications in
high-throughput MS2 experiments and show that an average $R^2$ of 80\% for
novel compounds can be achieved across 10 chemical properties prioritized by
medicinal chemists. We use dimensionality reduction techniques and experiments
with different floating point resolutions to show the essential role
multi-scale sinusoidal embeddings play in learning from MS2 data.
( 2
min )
Motivated by the recent success of Machine Learning tools in wireless
communications, the idea of semantic communication by Weaver from 1949 has
gained attention. It breaks with Shannon's classic design paradigm by aiming to
transmit the meaning, i.e., semantics, of a message instead of its exact
version, allowing for information rate savings. In this work, we apply the
Stochastic Policy Gradient (SPG) to design a semantic communication system by
reinforcement learning, not requiring a known or differentiable channel model -
a crucial step towards deployment in practice. Further, we motivate the use of
SPG for both classic and semantic communication from the maximization of the
mutual information between received and target variables. Numerical results
show that our approach achieves comparable performance to a model-aware
approach based on the reparametrization trick, albeit with a decreased
convergence rate.
( 2
min )
This paper considers the problem of helping humans exercise scalable
oversight over deep neural networks (DNNs). Adversarial examples can be useful
by helping to reveal weaknesses in DNNs, but they can be difficult to interpret
or draw actionable conclusions from. Some previous works have proposed using
human-interpretable adversarial attacks including copy/paste attacks in which
one natural image pasted into another causes an unexpected misclassification.
We build on these with two contributions. First, we introduce Search for
Natural Adversarial Features Using Embeddings (SNAFUE) which offers a fully
automated method for finding copy/paste attacks. Second, we use SNAFUE to red
team an ImageNet classifier. We reproduce copy/paste attacks from previous
works and find hundreds of other easily-describable vulnerabilities, all
without a human in the loop. Code is available at
https://github.com/thestephencasper/snafue
( 2
min )
We propose a novel graph-regularized neural network (GRNN) algorithm for tree
species classification. The proposed algorithm encompasses superpixel-based
segmentation for graph construction, a pixel-wise neural network classifier,
and the label propagation technique to generate an accurate and realistic
(emulating tree crowns) classification map on a sparsely annotated data set.
GRNN outperforms several state-of-the-art techniques not only for the standard
Indian Pines HSI but also achieves a high classification accuracy (approx. 92%)
on a new HSI data set collected over the heterogeneous forests of French Guiana
(FG) when less than 1% of the pixels are labeled. We further show that GRNN is
competitive with the state-of-the-art semi-supervised methods and exhibits a
small deviation in accuracy for different numbers of training samples and over
repeated trials with randomly sampled labeled pixels for training.
( 2
min )
Bayesian hierarchical mixture clustering (BHMC) improves traditionalBayesian
hierarchical clustering by replacing conventional Gaussian-to-Gaussian kernels
with a Hierarchical Dirichlet Process Mixture Model(HDPMM) for parent-to-child
diffusion in the generative process. However,BHMC may produce trees with high
nodal variance, indicating weak separation between nodes at higher levels. To
address this issue, we employ Posterior Regularization, which imposes
max-margin constraints on nodes at every level to enhance cluster separation.
We illustrate how to apply PR toBHMC and demonstrate its effectiveness in
improving the BHMC model.
( 2
min )
Strong demand for autonomous vehicles and the wide availability of 3D sensors
are continuously fueling the proposal of novel methods for 3D object detection.
In this paper, we provide a comprehensive survey of recent developments from
2012-2021 in 3D object detection covering the full pipeline from input data,
over data representation and feature extraction to the actual detection
modules. We introduce fundamental concepts, focus on a broad range of different
approaches that have emerged over the past decade, and propose a
systematization that provides a practical framework for comparing these
approaches with the goal of guiding future development, evaluation and
application activities. Specifically, our survey and systematization of 3D
object detection models and methods can help researchers and practitioners to
get a quick overview of the field by decomposing 3DOD solutions into more
manageable pieces.
( 2
min )
Photoplethysmogram (PPG) signals are easily contaminated by motion artifacts
in real-world settings, despite their widespread use in Internet-of-Things
(IoT) based wearable and smart health devices for cardiovascular health
monitoring. This study proposed a lightweight deep neural network, called
Tiny-PPG, for accurate and real-time PPG artifact segmentation on IoT edge
devices. The model was trained and tested on a public dataset, PPG DaLiA, which
featured complex artifacts with diverse lengths and morphologies during various
daily activities of 15 subjects using a watch-type device (Empatica E4). The
model structure, training method and loss function were specifically designed
to balance detection accuracy and speed for real-time PPG artifact detection in
resource-constrained embedded devices. To optimize the model size and
capability in multi-scale feature representation, the model employed deep
separable convolution and atrous spatial pyramid pooling modules, respectively.
Additionally, the contrastive loss was also utilized to further optimize the
feature embeddings. With additional model pruning, Tiny-PPG achieved
state-of-the-art detection accuracy of 87.8% while only having 19,726 model
parameters (0.15 megabytes), and was successfully deployed on an STM32 embedded
system for real-time PPG artifact detection. Therefore, this study provides an
effective solution for resource-constraint IoT smart health devices in PPG
artifact detection.
( 2
min )
Natural language generation from structured data mainly focuses on
surface-level descriptions, suffering from uncontrollable content selection and
low fidelity. Previous works leverage logical forms to facilitate logical
knowledge-conditioned text generation. Though achieving remarkable progress,
they are data-hungry, which makes the adoption for real-world applications
challenging with limited data. To this end, this paper proposes a unified
framework for logical knowledge-conditioned text generation in the few-shot
setting. With only a few seeds logical forms (e.g., 20/100 shot), our approach
leverages self-training and samples pseudo logical forms based on content and
structure consistency. Experimental results demonstrate that our approach can
obtain better few-shot performance than baselines.
( 2
min )
Latent variable modeling in non-autoregressive neural machine translation
(NAT) is a promising approach to mitigate the multimodality problem. In the
previous works, they added an auxiliary model to estimate the posterior
distribution of the latent variable conditioned on the source and target
sentences. However, it causes several disadvantages, such as redundant
information extraction in the latent variable, increasing parameters, and a
tendency to ignore a part of the information from the inputs. In this paper, we
propose a new latent variable modeling that is based on a dual reconstruction
perspective and an advanced hierarchical latent modeling approach. Our proposed
method, {\em LadderNMT}, shares a latent space across both languages so that it
hypothetically alleviates or solves the above disadvantages. Experimental
results quantitatively and qualitatively demonstrate that our proposed latent
variable modeling learns an advantageous latent space and significantly
improves translation quality in WMT translation tasks.
( 2
min )
Achieving resource efficiency while preserving end-user experience is
non-trivial for cloud application operators. As cloud applications
progressively adopt microservices, resource managers are faced with two
distinct levels of system behavior: the end-to-end application latency and
per-service resource usage. Translation between these two levels, however, is
challenging because user requests traverse heterogeneous services that
collectively (but unevenly) contribute to the end-to-end latency. This paper
presents Autothrottle, a bi-level learning-assisted resource management
framework for SLO-targeted microservices. It architecturally decouples
mechanisms of application SLO feedback and service resource control, and
bridges them with the notion of performance targets. This decoupling enables
targeted control policies for these two mechanisms, where we combine
lightweight heuristics and learning techniques. We evaluate Autothrottle on
three microservice applications, with workload traces from production
scenarios. Results show its superior CPU resource saving, up to 26.21% over the
best-performing baseline, and up to 93.84% over all baselines.
( 2
min )
This document presents some early explorations of applying Softly Masked
Language Modelling (SMLM) to symbolic music generation. SMLM can be seen as a
generalisation of masked language modelling (MLM), where instead of each
element of the input set being either known or unknown, elements can be partly
known. We demonstrate some results of applying SMLM to constrained symbolic
music generation using a transformer encoder architecture. Several audio
examples are available at https://erl-j.github.io/smlm-web-supplement/
( 2
min )
ChatGPT is another large language model (LLM) inline but due to its
performance and ability to converse effectively, it has gained a huge
popularity amongst research as well as industrial community. Recently, many
studies have been published to show the effectiveness, efficiency, integration,
and sentiments of chatGPT and other LLMs. In contrast, this study focuses on
the important aspects that are mostly overlooked, i.e. sustainability, privacy,
digital divide, and ethics and suggests that not only chatGPT but every
subsequent entry in the category of conversational bots should undergo
Sustainability, PrivAcy, Digital divide, and Ethics (SPADE) evaluation. This
paper discusses in detail about the issues and concerns raised over chatGPT in
line with aforementioned characteristics. We support our hypothesis by some
preliminary data collection and visualizations along with hypothesized facts.
We also suggest mitigations and recommendations for each of the concerns.
Furthermore, we also suggest some policies and recommendations for AI policy
act, if designed by the governments.
( 2
min )
In recent years, multitudes of researches have applied deep learning to
automatic sleep stage classification. Whereas actually, these works have paid
less attention to the issue of cross-subject in sleep staging. At the same
time, emerging neuroscience theories on inter-subject correlations can provide
new insights for cross-subject analysis. This paper presents the MViTime model
that have been used in sleep staging study. And we implement the inter-subject
correlation theory through contrastive learning, providing a feasible solution
to address the cross-subject problem in sleep stage classification. Finally,
experimental results and conclusions are presented, demonstrating that the
developed method has achieved state-of-the-art performance on sleep staging.
The results of the ablation experiment also demonstrate the effectiveness of
the cross-subject approach based on contrastive learning.
( 2
min )
We use explainable neural networks to connect the evolutionary history of
dark matter halos with their density profiles. The network captures independent
factors of variation in the density profiles within a low-dimensional
representation, which we physically interpret using mutual information. Without
any prior knowledge of the halos' evolution, the network recovers the known
relation between the early time assembly and the inner profile, and discovers
that the profile beyond the virial radius is described by a single parameter
capturing the most recent mass accretion rate. The results illustrate the
potential for machine-assisted scientific discovery in complicated
astrophysical datasets.
( 2
min )
Selecting a minimal feature set that is maximally informative about a target
variable is a central task in machine learning and statistics. Information
theory provides a powerful framework for formulating feature selection
algorithms -- yet, a rigorous, information-theoretic definition of feature
relevancy, which accounts for feature interactions such as redundant and
synergistic contributions, is still missing. We argue that this lack is
inherent to classical information theory which does not provide measures to
decompose the information a set of variables provides about a target into
unique, redundant, and synergistic contributions. Such a decomposition has been
introduced only recently by the partial information decomposition (PID)
framework. Using PID, we clarify why feature selection is a conceptually
difficult problem when approached using information theory and provide a novel
definition of feature relevancy and redundancy in PID terms. From this
definition, we show that the conditional mutual information (CMI) maximizes
relevancy while minimizing redundancy and propose an iterative, CMI-based
algorithm for practical feature selection. We demonstrate the power of our
CMI-based algorithm in comparison to the unconditional mutual information on
benchmark examples and provide corresponding PID estimates to highlight how PID
allows to quantify information contribution of features and their interactions
in feature-selection problems.
( 3
min )
Generating explanations for reinforcement learning (RL) is challenging as
actions may produce long-term effects on the future. In this paper, we develop
a novel framework for explainable RL by learning a causal world model without
prior knowledge of the causal structure of the environment. The model captures
the influence of actions, allowing us to interpret the long-term effects of
actions through causal chains, which present how actions influence
environmental variables and finally lead to rewards. Different from most
explanatory models which suffer from low accuracy, our model remains accurate
while improving explainability, making it applicable in model-based learning.
As a result, we demonstrate that our causal model can serve as the bridge
between explainability and learning.
( 2
min )
Since its inception, the field of unbiased learning to rank (ULTR) has
remained very active and has seen several impactful advancements in recent
years. This tutorial provides both an introduction to the core concepts of the
field and an overview of recent advancements in its foundations along with
several applications of its methods. The tutorial is divided into four parts:
Firstly, we give an overview of the different forms of bias that can be
addressed with ULTR methods. Secondly, we present a comprehensive discussion of
the latest estimation techniques in the ULTR field. Thirdly, we survey
published results of ULTR in real-world applications. Fourthly, we discuss the
connection between ULTR and fairness in ranking. We end by briefly reflecting
on the future of ULTR research and its applications. This tutorial is intended
to benefit both researchers and industry practitioners who are interested in
developing new ULTR solutions or utilizing them in real-world applications.
( 2
min )
Detecting plagiarism involves finding similar items in two different sources.
In this article, we propose a novel method for detecting plagiarism that is
based on attention mechanism-based long short-term memory (LSTM) and
bidirectional encoder representations from transformers (BERT) word embedding,
enhanced with optimized differential evolution (DE) method for pre-training and
a focal loss function for training. BERT could be included in a downstream task
and fine-tuned as a task-specific BERT can be included in a downstream task and
fine-tuned as a task-specific structure, while the trained BERT model is
capable of detecting various linguistic characteristics. Unbalanced
classification is one of the primary issues with plagiarism detection. We
suggest a focal loss-based training technique that carefully learns minority
class instances to solve this. Another issue that we tackle is the training
phase itself, which typically employs gradient-based methods like
back-propagation for the learning process and thus suffers from some drawbacks,
including sensitivity to initialization. To initiate the BP process, we suggest
a novel DE algorithm that makes use of a clustering-based mutation operator.
Here, a winning cluster is identified for the current DE population, and a
fresh updating method is used to produce potential answers. We evaluate our
proposed approach on three benchmark datasets ( MSRP, SNLI, and SemEval2014)
and demonstrate that it performs well when compared to both conventional and
population-based methods.
( 3
min )
This work investigates pretrained audio representations for few shot Sound
Event Detection. We specifically address the task of few shot detection of
novel acoustic sequences, or sound events with semantically meaningful temporal
structure, without assuming access to non-target audio. We develop procedures
for pretraining suitable representations, and methods which transfer them to
our few shot learning scenario. Our experiments evaluate the general purpose
utility of our pretrained representations on AudioSet, and the utility of
proposed few shot methods via tasks constructed from real-world acoustic
sequences. Our pretrained embeddings are suitable to the proposed task, and
enable multiple aspects of our few shot framework.
( 2
min )
Recent years have witnessed the proliferation of traffic accidents, which led
wide researches on Automated Vehicle (AV) technologies to reduce vehicle
accidents, especially on risk assessment framework of AV technologies. However,
existing time-based frameworks can not handle complex traffic scenarios and
ignore the motion tendency influence of each moving objects on the risk
distribution, leading to performance degradation. To address this problem, we
novelly propose a comprehensive driving risk management framework named RCP-RF
based on potential field theory under Connected and Automated Vehicles (CAV)
environment, where the pedestrian risk metric are combined into a unified
road-vehicle driving risk management framework. Different from existing
algorithms, the motion tendency between ego and obstacle cars and the
pedestrian factor are legitimately considered in the proposed framework, which
can improve the performance of the driving risk model. Moreover, it requires
only O(N 2) of time complexity in the proposed method. Empirical studies
validate the superiority of our proposed framework against state-of-the-art
methods on real-world dataset NGSIM and real AV platform.
( 2
min )
We study principal component analysis (PCA), where given a dataset in
$\mathbb{R}^d$ from a distribution, the task is to find a unit vector $v$ that
approximately maximizes the variance of the distribution after being projected
along $v$. Despite being a classical task, standard estimators fail drastically
if the data contains even a small fraction of outliers, motivating the problem
of robust PCA. Recent work has developed computationally-efficient algorithms
for robust PCA that either take super-linear time or have sub-optimal error
guarantees. Our main contribution is to develop a nearly-linear time algorithm
for robust PCA with near-optimal error guarantees. We also develop a
single-pass streaming algorithm for robust PCA with memory usage nearly-linear
in the dimension.
( 2
min )
Analysis of Electrochemical Impedance Spectroscopy (EIS) data for
electrochemical systems often consists of defining an Equivalent Circuit Model
(ECM) using expert knowledge and then optimizing the model parameters to
deconvolute various resistance, capacitive, inductive, or diffusion responses.
For small data sets, this procedure can be conducted manually; however, it is
not feasible to manually define a proper ECM for extensive data sets with a
wide range of EIS responses. Automatic identification of an ECM would
substantially accelerate the analysis of large sets of EIS data. We showcase
machine learning methods to classify the ECMs of 9,300 impedance spectra
provided by QuantumScape for the BatteryDEV hackathon. The best-performing
approach is a gradient-boosted tree model utilizing a library to automatically
generate features, followed by a random forest model using the raw spectral
data. A convolutional neural network using boolean images of Nyquist
representations is presented as an alternative, although it achieves a lower
accuracy. We publish the data and open source the associated code. The
approaches described in this article can serve as benchmarks for further
studies. A key remaining challenge is the identifiability of the labels,
underlined by the model performances and the comparison of misclassified
spectra.
( 3
min )
We provide a psychometric-grounded exposition of bias and fairness as applied
to a typical machine learning pipeline for affective computing. We expand on an
interpersonal communication framework to elucidate how to identify sources of
bias that may arise in the process of inferring human emotions and other
psychological constructs from observed behavior. Various methods and metrics
for measuring fairness and bias are discussed along with pertinent implications
within the United States legal context. We illustrate how to measure some types
of bias and fairness in a case study involving automatic personality and
hireability inference from multimodal data collected in video interviews for
mock job applications. We encourage affective computing researchers and
practitioners to encapsulate bias and fairness in their research processes and
products and to consider their role, agency, and responsibility in promoting
equitable and just systems.
( 2
min )
This paper evaluates the viability of using fixed language models for
training text classification networks on low-end hardware. We combine language
models with a CNN architecture and put together a comprehensive benchmark with
8 datasets covering single-label and multi-label classification of topic,
sentiment, and genre. Our observations are distilled into a list of trade-offs,
concluding that there are scenarios, where not fine-tuning a language model
yields competitive effectiveness at faster training, requiring only a quarter
of the memory compared to fine-tuning.
( 2
min )
In this work, we propose a novel evolutionary algorithm for neural
architecture search, applicable to global search spaces. The algorithm's
architectural representation organizes the topology in multiple hierarchical
modules, while the design process exploits this representation, in order to
explore the search space. We also employ a curation system, which promotes the
utilization of well performing sub-structures to subsequent generations. We
apply our method to Fashion-MNIST and NAS-Bench101, achieving accuracies of
$93.2\%$ and $94.8\%$ respectively in a relatively small number of generations.
( 2
min )
Many machine learning (ML) libraries are accessible online for ML
practitioners. Typical ML pipelines are complex and consist of a series of
steps, each of them invoking several ML libraries. In this demo paper, we
present ExeKGLib, a Python library that allows users with coding skills and
minimal ML knowledge to build ML pipelines. ExeKGLib relies on knowledge graphs
to improve the transparency and reusability of the built ML workflows, and to
ensure that they are executable. We demonstrate the usage of ExeKGLib and
compare it with conventional ML code to show its benefits.
( 2
min )
Numerical models are used widely for parameter reconstructions in the field
of optical nano metrology. To obtain geometrical parameters of a nano
structured line grating, we fit a finite element numerical model to an
experimental data set by using the Bayesian target vector optimization method.
Gaussian process surrogate models are trained during the reconstruction.
Afterwards, we employ a Markov chain Monte Carlo sampler on the surrogate
models to determine the full model parameter distribution for the reconstructed
model parameters. The choice of numerical discretization parameters, like the
polynomial order of the finite element ansatz functions, impacts the numerical
discretization error of the forward model. In this study we investigate the
impact of numerical discretization parameters of the forward problem on the
reconstructed parameters as well as on the model parameter distributions. We
show that such a convergence study allows to determine numerical parameters
which allow for efficient and accurate reconstruction results.
( 2
min )
We describe how interpretable boosting algorithms based on ridge-regularized
generalized linear models can be used to analyze high-dimensional environmental
data. We illustrate this by using environmental, social, human and biophysical
data to predict the financial vulnerability of farmers in Chile and Tunisia
against climate hazards. We show how group structures can be considered and how
interactions can be found in high-dimensional datasets using a novel 2-step
boosting approach. The advantages and efficacy of the proposed method are shown
and discussed. Results indicate that the presence of interaction effects only
improves predictive power when included in two-step boosting. The most
important variable in predicting all types of vulnerabilities are natural
assets. Other important variables are the type of irrigation, economic assets
and the presence of crop damage of near farms.
( 2
min )
Random graphs are increasingly becoming objects of interest for modeling
networks in a wide range of applications. Latent position random graph models
posit that each node is associated with a latent position vector, and that
these vectors follow some geometric structure in the latent space. In this
paper, we consider random dot product graphs, in which an edge is formed
between two nodes with probability given by the inner product of their
respective latent positions. We assume that the latent position vectors lie on
an unknown one-dimensional curve and are coupled with a response covariate via
a regression model. Using the geometry of the underlying latent position
vectors, we propose a manifold learning and graph embedding technique to
predict the response variable on out-of-sample nodes, and we establish
convergence guarantees for these responses. Our theoretical results are
supported by simulations and an application to Drosophila brain data.
( 2
min )
This paper formulates a general cross validation framework for signal
denoising. The general framework is then applied to nonparametric regression
methods such as Trend Filtering and Dyadic CART. The resulting cross validated
versions are then shown to attain nearly the same rates of convergence as are
known for the optimally tuned analogues. There did not exist any previous
theoretical analyses of cross validated versions of Trend Filtering or Dyadic
CART. To illustrate the generality of the framework we also propose and study
cross validated versions of two fundamental estimators; lasso for high
dimensional linear regression and singular value thresholding for matrix
estimation. Our general framework is inspired by the ideas in Chatterjee and
Jafarov (2015) and is potentially applicable to a wide range of estimation
methods which use tuning parameters.
( 2
min )
This paper provide several mathematical analyses of the diffusion model in
machine learning. The drift term of the backwards sampling process is
represented as a conditional expectation involving the data distribution and
the forward diffusion. The training process aims to find such a drift function
by minimizing the mean-squared residue related to the conditional expectation.
Using small-time approximations of the Green's function of the forward
diffusion, we show that the analytical mean drift function in DDPM and the
score function in SGM asymptotically blow up in the final stages of the
sampling process for singular data distributions such as those concentrated on
lower-dimensional manifolds, and is therefore difficult to approximate by a
network. To overcome this difficulty, we derive a new target function and
associated loss, which remains bounded even for singular data distributions. We
illustrate the theoretical findings with several numerical examples.
( 2
min )
We study principal component analysis (PCA), where given a dataset in
$\mathbb{R}^d$ from a distribution, the task is to find a unit vector $v$ that
approximately maximizes the variance of the distribution after being projected
along $v$. Despite being a classical task, standard estimators fail drastically
if the data contains even a small fraction of outliers, motivating the problem
of robust PCA. Recent work has developed computationally-efficient algorithms
for robust PCA that either take super-linear time or have sub-optimal error
guarantees. Our main contribution is to develop a nearly-linear time algorithm
for robust PCA with near-optimal error guarantees. We also develop a
single-pass streaming algorithm for robust PCA with memory usage nearly-linear
in the dimension.
( 2
min )
Clustering is at the very core of machine learning, and its applications
proliferate with the increasing availability of data. However, as datasets
grow, comparing clusterings with an adjustment for chance becomes
computationally difficult, preventing unbiased ground-truth comparisons and
solution selection. We propose FastAMI, a Monte Carlo-based method to
efficiently approximate the Adjusted Mutual Information (AMI) and extend it to
the Standardized Mutual Information (SMI). The approach is compared with the
exact calculation and a recently developed variant of the AMI based on pairwise
permutations, using both synthetic and real data. In contrast to the exact
calculation our method is fast enough to enable these adjusted
information-theoretic comparisons for large datasets while maintaining
considerably more accurate results than the pairwise approach.
( 2
min )
The analysis of large-scale time-series network data, such as social media
and email communications, remains a significant challenge for graph analysis
methodology. In particular, the scalability of graph analysis is a critical
issue hindering further progress in large-scale downstream inference. In this
paper, we introduce a novel approach called "temporal encoder embedding" that
can efficiently embed large amounts of graph data with linear complexity. We
apply this method to an anonymized time-series communication network from a
large organization spanning 2019-2020, consisting of over 100 thousand vertices
and 80 million edges. Our method embeds the data within 10 seconds on a
standard computer and enables the detection of communication pattern shifts for
individual vertices, vertex communities, and the overall graph structure.
Through supporting theory and synthesis studies, we demonstrate the theoretical
soundness of our approach under random graph models and its numerical
effectiveness through simulation studies.
( 2
min )
Jeff Wilke SM '93, former CEO of Amazon’s Worldwide Consumer business, brings his LGO playbook to his new mission of revitalizing manufacturing in the U.S.
( 12
min )
In this post, we discuss a machine learning (ML) solution for complex image searches using Amazon Kendra and Amazon Rekognition. Specifically, we use the example of architecture diagrams for complex images due to their incorporation of numerous different visual icons and text. With the internet, searching and obtaining an image has never been easier. Most […]
( 17
min )
This is a joint post co-written by AWS and Voxel51. Voxel51 is the company behind FiftyOne, the open-source toolkit for building high-quality datasets and computer vision models. A retail company is building a mobile app to help customers buy clothes. To create this app, they need a high-quality dataset containing clothing images, labeled with different […]
( 16
min )
AI Weirdness: the strange side of machine learning
( 2
min )
Automated revenue cycle management (RCM) is becoming an increasingly vital component in the healthcare industry, streamlining and accurately processing complex billing tasks. Using artificial intelligence (AI) and machine learning capabilities will help in ensuring accuracy, minimize human error, and free up personnel to focus on more important tasks. In addition, utilizing AI for RCM will… Read More »Maximizing Revenue in Psychology Practices: Leveraging AI for Billing Optimization
The post Maximizing Revenue in Psychology Practices: Leveraging AI for Billing Optimization appeared first on Data Science Central.
( 21
min )
The world of artificial intelligence (AI) and machine learning (ML) has been witnessing a paradigm shift with the rise of generative AI models that can create human-like text, images, code, and audio. Compared to classical ML models, generative AI models are significantly bigger and more complex. However, their increasing complexity also comes with high costs […]
( 12
min )
Time series forecasting refers to the process of predicting future values of time series data (data that is collected at regular intervals over time). Simple methods for time series forecasting use historical values of the same variable whose future values need to be predicted, whereas more complex, machine learning (ML)-based methods use additional information, such […]
( 16
min )
Generative AI is gaining a lot of public attention at present, with talk around products such as GPT4, ChatGPT, DALL-E2, Bard, and many other AI technologies. Many customers have been asking for more information on AWS’s generative AI solutions. The aim of this post is to address those needs. This post provides an overview of […]
( 10
min )
Diffusion models have been used to generate photorealistic images and short videos, compose music, and synthesize speech. In a new paper, Microsoft Researchers explore how they can be used to imitate human behavior in interactive environments.
The post Using generative AI to imitate human behavior appeared first on Microsoft Research.
( 11
min )
Diffusion models have been used to generate photorealistic images and short videos, compose music, and synthesize speech. In a new paper, Microsoft Researchers explore how they can be used to imitate human behavior in interactive environments.
The post Using generative AI to imitate human behavior appeared first on Microsoft Research.
( 11
min )
Kris Kersey is an embedded software developer with over 20 years of experience, an educational YouTuber with 30,000+ subscribers, and a lifelong lover of comics and cosplay. These interests and expertise came together in his first-ever project using the NVIDIA Jetson platform for edge AI and robotics when he created a fully functional superhero helmet Read article >
( 6
min )
What has it got in its pocketses? More games coming in May, that’s what. GFN Thursday gets the summer started early with two newly supported games this week and 16 more coming later this month — including The Lord of the Rings: Gollum. Don’t forget to take advantage of the limited-time discount on six-month Priority Read article >
( 6
min )
The system they developed eliminates a source of bias in simulations, leading to improved algorithms that can boost the performance of applications.
( 9
min )
eXplainable artificial intelligence (XAI) methods have emerged to convert the
black box of machine learning models into a more digestible form. These methods
help to communicate how the model works with the aim of making machine learning
models more transparent and increasing the trust of end-users into their
output. SHapley Additive exPlanations (SHAP) and Local Interpretable Model
Agnostic Explanation (LIME) are two widely used XAI methods particularly with
tabular data. In this commentary piece, we discuss the way the explainability
metrics of these two methods are generated and propose a framework for
interpretation of their outputs, highlighting their weaknesses and strengths.
( 2
min )
This paper describes our submission to the MEDIQA-Chat 2023 shared task for
automatic clinical note generation from doctor-patient conversations. We report
results for two approaches: the first fine-tunes a pre-trained language model
(PLM) on the shared task data, and the second uses few-shot in-context learning
(ICL) with a large language model (LLM). Both achieve high performance as
measured by automatic metrics (e.g. ROUGE, BERTScore) and ranked second and
first, respectively, of all submissions to the shared task. Expert human
scrutiny indicates that notes generated via the ICL-based approach with GPT-4
are preferred about as often as human-written notes, making it a promising path
toward automated note generation from doctor-patient conversations.
( 2
min )
Contrastively trained encoders have recently been proven to invert the
data-generating process: they encode each input, e.g., an image, into the true
latent vector that generated the image (Zimmermann et al., 2021). However,
real-world observations often have inherent ambiguities. For instance, images
may be blurred or only show a 2D view of a 3D object, so multiple latents could
have generated them. This makes the true posterior for the latent vector
probabilistic with heteroscedastic uncertainty. In this setup, we extend the
common InfoNCE objective and encoders to predict latent distributions instead
of points. We prove that these distributions recover the correct posteriors of
the data-generating process, including its level of aleatoric uncertainty, up
to a rotation of the latent space. In addition to providing calibrated
uncertainty estimates, these posteriors allow the computation of credible
intervals in image retrieval. They comprise images with the same latent as a
given query, subject to its uncertainty. Code is available at
https://github.com/mkirchhof/Probabilistic_Contrastive_Learning
( 2
min )
Differentiable particle filters are an emerging class of particle filtering
methods that use neural networks to construct and learn parametric state-space
models. In real-world applications, both the state dynamics and measurements
can switch between a set of candidate models. For instance, in target tracking,
vehicles can idle, move through traffic, or cruise on motorways, and
measurements are collected in different geographical or weather conditions.
This paper proposes a new differentiable particle filter for regime-switching
state-space models. The method can learn a set of unknown candidate dynamic and
measurement models and track the state posteriors. We evaluate the performance
of the novel algorithm in relevant models, showing its great performance
compared to other competitive algorithms.
( 2
min )
We present a new method for functional tissue unit segmentation at the
cellular level, which utilizes the latest deep learning semantic segmentation
approaches together with domain adaptation and semi-supervised learning
techniques. This approach allows for minimizing the domain gap, class
imbalance, and captures settings influence between HPA and HubMAP datasets. The
presented approach achieves comparable with state-of-the-art-result in
functional tissue unit segmentation at the cellular level. The source code is
available at https://github.com/VSydorskyy/hubmap_2022_htt_solution
( 2
min )
Prior research has investigated the impact of various linguistic features on
cross-lingual transfer performance. In this study, we investigate the manner in
which this effect can be mapped onto the representation space. While past
studies have focused on the impact on cross-lingual alignment in multilingual
language models during fine-tuning, this study examines the absolute evolution
of the respective language representation spaces produced by MLLMs. We place a
specific emphasis on the role of linguistic characteristics and investigate
their inter-correlation with the impact on representation spaces and
cross-lingual transfer performance. Additionally, this paper provides
preliminary evidence of how these findings can be leveraged to enhance transfer
to linguistically distant languages.
( 2
min )
We present a study using new computational methods, based on a novel
combination of machine learning for inferring admixture hidden Markov models
and probabilistic model checking, to uncover interaction styles in a mobile
app. These styles are then used to inform a redesign, which is implemented,
deployed, and then analysed using the same methods. The data sets are logged
user traces, collected over two six-month deployments of each version,
involving thousands of users and segmented into different time intervals. The
methods do not assume tasks or absolute metrics such as measures of engagement,
but uncover the styles through unsupervised inference of clusters and analysis
with probabilistic temporal logic. For both versions there was a clear
distinction between the styles adopted by users during the first day/week/month
of usage, and during the second and third months, a result we had not
anticipated.
( 2
min )
Data valuation using Shapley value has emerged as a prevalent research domain
in machine learning applications. However, it is a challenge to address the
role of order in data cooperation as most research lacks such discussion. To
tackle this problem, this paper studies the definition of the partial ordinal
Shapley value by group theory in abstract algebra. Besides, since the
calculation of the partial ordinal Shapley value requires exponential time,
this paper also gives three algorithms for approximating the results. The
Truncated Monte Carlo algorithm is derived from the classic Shapley value
approximation algorithm. The Classification Monte Carlo algorithm and the
Classification Truncated Monte Carlo algorithm are based on the fact that the
data points in the same class provide similar information, then we can
accelerate the calculation by leaving out some data points in each class.
( 2
min )
Respiratory syncytial virus (RSV) is one of the most dangerous respiratory
diseases for infants and young children. Due to the nonpharmaceutical
intervention (NPI) imposed in the COVID-19 outbreak, the seasonal transmission
pattern of RSV has been discontinued in 2020 and then shifted months ahead in
2021 in the northern hemisphere. It is critical to understand how COVID-19
impacts RSV and build predictive algorithms to forecast the timing and
intensity of RSV reemergence in post-COVID-19 seasons. In this paper, we
propose a deep coupled tensor factorization machine, dubbed as DeCom, for post
COVID-19 RSV prediction. DeCom leverages tensor factorization and residual
modeling. It enables us to learn the disrupted RSV transmission reliably under
COVID-19 by taking both the regular seasonal RSV transmission pattern and the
NPI into consideration. Experimental results on a real RSV dataset show that
DeCom is more accurate than the state-of-the-art RSV prediction algorithms and
achieves up to 46% lower root mean square error and 49% lower mean absolute
error for country-level prediction compared to the baselines.
( 2
min )
In an effort to address the training instabilities of GANs, we introduce a
class of dual-objective GANs with different value functions (objectives) for
the generator (G) and discriminator (D). In particular, we model each objective
using $\alpha$-loss, a tunable classification loss, to obtain
$(\alpha_D,\alpha_G)$-GANs, parameterized by $(\alpha_D,\alpha_G)\in
(0,\infty]^2$. For sufficiently large number of samples and capacities for G
and D, we show that the resulting non-zero sum game simplifies to minimizing an
$f$-divergence under appropriate conditions on $(\alpha_D,\alpha_G)$. In the
finite sample and capacity setting, we define estimation error to quantify the
gap in the generator's performance relative to the optimal setting with
infinite samples and obtain upper bounds on this error, showing it to be order
optimal under certain conditions. Finally, we highlight the value of tuning
$(\alpha_D,\alpha_G)$ in alleviating training instabilities for the synthetic
2D Gaussian mixture ring and the Stacked MNIST datasets.
( 2
min )
An old problem in multivariate statistics is that linear Gaussian models are
often unidentifiable, i.e. some parameters cannot be uniquely estimated. In
factor (component) analysis, an orthogonal rotation of the factors is
unidentifiable, while in linear regression, the direction of effect cannot be
identified. For such linear models, non-Gaussianity of the (latent) variables
has been shown to provide identifiability. In the case of factor analysis, this
leads to independent component analysis, while in the case of the direction of
effect, non-Gaussian versions of structural equation modelling solve the
problem. More recently, we have shown how even general nonparametric nonlinear
versions of such models can be estimated. Non-Gaussianity is not enough in this
case, but assuming we have time series, or that the distributions are suitably
modulated by some observed auxiliary variables, the models are identifiable.
This paper reviews the identifiability theory for the linear and nonlinear
cases, considering both factor analytic models and structural equation models.
( 2
min )
Contrastively trained encoders have recently been proven to invert the
data-generating process: they encode each input, e.g., an image, into the true
latent vector that generated the image (Zimmermann et al., 2021). However,
real-world observations often have inherent ambiguities. For instance, images
may be blurred or only show a 2D view of a 3D object, so multiple latents could
have generated them. This makes the true posterior for the latent vector
probabilistic with heteroscedastic uncertainty. In this setup, we extend the
common InfoNCE objective and encoders to predict latent distributions instead
of points. We prove that these distributions recover the correct posteriors of
the data-generating process, including its level of aleatoric uncertainty, up
to a rotation of the latent space. In addition to providing calibrated
uncertainty estimates, these posteriors allow the computation of credible
intervals in image retrieval. They comprise images with the same latent as a
given query, subject to its uncertainty. Code is available at
https://github.com/mkirchhof/Probabilistic_Contrastive_Learning
( 2
min )
eXplainable artificial intelligence (XAI) methods have emerged to convert the
black box of machine learning models into a more digestible form. These methods
help to communicate how the model works with the aim of making machine learning
models more transparent and increasing the trust of end-users into their
output. SHapley Additive exPlanations (SHAP) and Local Interpretable Model
Agnostic Explanation (LIME) are two widely used XAI methods particularly with
tabular data. In this commentary piece, we discuss the way the explainability
metrics of these two methods are generated and propose a framework for
interpretation of their outputs, highlighting their weaknesses and strengths.
( 2
min )
We establish matching upper and lower generalization error bounds for
mini-batch Gradient Descent (GD) training with either deterministic or
stochastic, data-independent, but otherwise arbitrary batch selection rules. We
consider smooth Lipschitz-convex/nonconvex/strongly-convex loss functions, and
show that classical upper bounds for Stochastic GD (SGD) also hold verbatim for
such arbitrary nonadaptive batch schedules, including all deterministic ones.
Further, for convex and strongly-convex losses we prove matching lower bounds
directly on the generalization error uniform over the aforementioned class of
batch schedules, showing that all such batch schedules generalize optimally.
Lastly, for smooth (non-Lipschitz) nonconvex losses, we show that full-batch
(deterministic) GD is essentially optimal, among all possible batch schedules
within the considered class, including all stochastic ones.
( 2
min )
We consider a general $p$-norm objective for experimental design problems
that captures some well-studied objectives (D/A/E-design) as special cases. We
prove that a randomized local search approach provides a unified algorithm to
solve this problem for all $p$. This provides the first approximation algorithm
for the general $p$-norm objective, and a nice interpolation of the best known
bounds of the special cases.
( 2
min )
The idea of adversarial learning of regularization functionals has recently
been introduced in the wider context of inverse problems. The intuition behind
this method is the realization that it is not only necessary to learn the basic
features that make up a class of signals one wants to represent, but also, or
even more so, which features to avoid in the representation. In this paper, we
will apply this approach to the problem of source separation by means of
non-negative matrix factorization (NMF) and present a new method for the
adversarial training of NMF bases. We show in numerical experiments, both for
image and audio separation, that this leads to a clear improvement of the
reconstructed signals, in particular in the case where little or no strong
supervision data is available.
( 2
min )
Respiratory syncytial virus (RSV) is one of the most dangerous respiratory
diseases for infants and young children. Due to the nonpharmaceutical
intervention (NPI) imposed in the COVID-19 outbreak, the seasonal transmission
pattern of RSV has been discontinued in 2020 and then shifted months ahead in
2021 in the northern hemisphere. It is critical to understand how COVID-19
impacts RSV and build predictive algorithms to forecast the timing and
intensity of RSV reemergence in post-COVID-19 seasons. In this paper, we
propose a deep coupled tensor factorization machine, dubbed as DeCom, for post
COVID-19 RSV prediction. DeCom leverages tensor factorization and residual
modeling. It enables us to learn the disrupted RSV transmission reliably under
COVID-19 by taking both the regular seasonal RSV transmission pattern and the
NPI into consideration. Experimental results on a real RSV dataset show that
DeCom is more accurate than the state-of-the-art RSV prediction algorithms and
achieves up to 46% lower root mean square error and 49% lower mean absolute
error for country-level prediction compared to the baselines.
( 2
min )
A collaborative research team from the MIT-Takeda Program combined physics and machine learning to characterize rough particle surfaces in pharmaceutical pills and powders.
( 8
min )
Generative AI (GenAI) and large language models (LLMs), such as those available soon via Amazon Bedrock and Amazon Titan are transforming the way developers and enterprises are able to solve traditionally complex challenges related to natural language processing and understanding. Some of the benefits offered by LLMs include the ability to create more capable and […]
( 12
min )
Amazon SageMaker Studio is the first fully integrated development environment (IDE) for ML. It provides a single, web-based visual interface where you can perform all machine learning (ML) development steps required to build, train, tune, debug, deploy, and monitor models. It gives data scientists all the tools you need to take ML models from experimentation […]
( 13
min )
New generations of CPUs offer a significant performance improvement in machine learning (ML) inference due to specialized built-in instructions. Combined with their flexibility, high speed of development, and low operating cost, these general-purpose processors offer an alternative to other existing hardware solutions. AWS, Arm, Meta and others helped optimize the performance of PyTorch 2.0 inference […]
( 6
min )
This post is co-written by Jyoti Sharma and Sharmo Sarkar from Vericast. For any machine learning (ML) problem, the data scientist begins by working with data. This includes gathering, exploring, and understanding the business and technical aspects of the data, along with evaluation of any manipulations that may be needed for the model building process. […]
( 13
min )
Collaboration is key to bringing ideas from lab to life. In the first episode of the #MSRPodcast series “Collaborators,” learn how GitHub’s Kasia Sitkiewicz and Protocol Labs’ Petar Maymounkov are teaming up to make open-source collaborative work better.
The post Collaborators: Gov4git with Petar Maymounkov and Kasia Sitkiewicz appeared first on Microsoft Research.
( 31
min )
This work lists and describes the main recent strategies for building
fixed-length, dense and distributed representations for words, based on the
distributional hypothesis. These representations are now commonly called word
embeddings and, in addition to encoding surprisingly good syntactic and
semantic information, have been proven useful as extra features in many
downstream NLP tasks.
( 2
min )
Adversarial training, which is to enhance robustness against adversarial
attacks, has received much attention because it is easy to generate
human-imperceptible perturbations of data to deceive a given deep neural
network. In this paper, we propose a new adversarial training algorithm that is
theoretically well motivated and empirically superior to other existing
algorithms. A novel feature of the proposed algorithm is to apply more
regularization to data vulnerable to adversarial attacks than other existing
regularization algorithms do. Theoretically, we show that our algorithm can be
understood as an algorithm of minimizing the regularized empirical risk
motivated from a newly derived upper bound of the robust risk. Numerical
experiments illustrate that our proposed algorithm improves the generalization
(accuracy on examples) and robustness (accuracy on adversarial attacks)
simultaneously to achieve the state-of-the-art performance.
( 2
min )
We introduce Robust Exploration via Clustering-based Online Density
Estimation (RECODE), a non-parametric method for novelty-based exploration that
estimates visitation counts for clusters of states based on their similarity in
a chosen embedding space. By adapting classical clustering to the nonstationary
setting of Deep RL, RECODE can efficiently track state visitation counts over
thousands of episodes. We further propose a novel generalization of the inverse
dynamics loss, which leverages masked transformer architectures for multi-step
prediction; which in conjunction with RECODE achieves a new state-of-the-art in
a suite of challenging 3D-exploration tasks in DM-Hard-8. RECODE also sets new
state-of-the-art in hard exploration Atari games, and is the first agent to
reach the end screen in "Pitfall!".
( 2
min )
Gaussian copula mixture models (GCMM) are the generalization of Gaussian
Mixture models using the concept of copula. Its mathematical definition is
given and the properties of likelihood function are studied in this paper.
Based on these properties, extended Expectation Maximum algorithms are
developed for estimating parameters for the mixture of copulas while marginal
distributions corresponding to each component is estimated using separate
nonparametric statistical methods. In the experiment, GCMM can achieve better
goodness-of-fitting given the same number of clusters as GMM; furthermore, GCMM
can utilize unsynchronized data on each dimension to achieve deeper mining of
data.
( 2
min )
AV1, the next-generation video codec, is expanding its reach with today’s release of OBS Studio 29.1. This latest software update adds support for AV1 streaming to YouTube over Enhanced RTMP. All GeForce RTX 40 Series GPUs — including laptop GPUs and the recently launched GeForce RTX 4070 — support real-time AV1 hardware encoding, providing 40% Read article >
( 5
min )
NVIDIA today introduced a wave of cutting-edge AI research that will enable developers and artists to bring their ideas to life — whether still or moving, in 2D or 3D, hyperrealistic or fantastical. Around 20 NVIDIA Research papers advancing generative AI and neural graphics — including collaborations with over a dozen universities in the U.S., Read article >
( 8
min )
Content creator Grant Abbitt embodies selflessness, one of the best qualities that a creative can possess. Passionate about giving back to the creative community, Abbitt offers inspiration, guidance and free education for others in his field through YouTube tutorials.
( 7
min )
One of the most popular models available today is XGBoost. With the ability to solve various problems such as classification and regression, XGBoost has become a popular option that also falls into the category of tree-based models. In this post, we dive deep to see how Amazon SageMaker can serve these models using NVIDIA Triton […]
( 18
min )
Machine learning (ML) helps organizations generate revenue, reduce costs, mitigate risk, drive efficiencies, and improve quality by optimizing core business functions across multiple business units such as marketing, manufacturing, operations, sales, finance, and customer service. With AWS ML, organizations can accelerate the value creation from months to days. Amazon SageMaker Canvas is a visual, point-and-click […]
( 8
min )
Today, we announce the availability of sample notebooks that demonstrate question answering tasks using a Retrieval Augmented Generation (RAG)-based approach with large language models (LLMs) in Amazon SageMaker JumpStart. Text generation using RAG with LLMs enables you to generate domain-specific text outputs by supplying specific external data as part of the context fed to LLMs. […]
( 13
min )
Announcements Big tech must weigh AI’s risks vs. rewards In an interview with the New York Times, Hinton noted the pace of AI advancement is far beyond what he and other tech experts predicted. Hinton said that Google acted very responsibly while he worked on its AI development efforts. His concerns are due to AI’s… Read More »DSC Weekly 2 May 2023 – Big tech must weigh AI’s risks vs. rewards
The post DSC Weekly 2 May 2023 – Big tech must weigh AI’s risks vs. rewards appeared first on Data Science Central.
( 19
min )
Discover the differences between AI, machine learning, and deep learning in this comprehensive guide. Learn how each technology works, their key applications, and the skills required for a career in data science.
The post AI vs Machine Learning vs Deep Learning appeared first on Data Science Central.
( 23
min )
The rapid adoption of smart phones and other mobile platforms has generated an enormous amount of image data. According to Gartner, unstructured data now represents 80–90% of all new enterprise data, but just 18% of organizations are taking advantage of this data. This is mainly due to a lack of expertise and the large amount […]
( 9
min )
Irene Politkoff, Founder and Chief Product Evangelist at semantic modeling tools provider TopQuadrant, posted this description of the large language model (LLM) ChatGPT: “ChatGPT doesn’t access a database of facts to answer your questions. Instead, its responses are based on patterns that it saw in the training data. So ChatGPT is not always trustworthy.” Georgetown… Read More »Can we boost the confidence scores of LLM answers with the help of knowledge graphs?
The post Can we boost the confidence scores of LLM answers with the help of knowledge graphs? appeared first on Data Science Central.
( 20
min )
Customers from Japan to Ecuador and Sweden are using NVIDIA DGX H100 systems like AI factories to manufacture intelligence. They’re creating services that offer AI-driven insights in finance, healthcare, law, IT and telecom — and working to transform their industries in the process. Among the dozens of use cases, one aims to predict how factory Read article >
( 6
min )
Textual backdoor attacks pose a practical threat to existing systems, as they
can compromise the model by inserting imperceptible triggers into inputs and
manipulating labels in the training dataset. With cutting-edge generative
models such as GPT-4 pushing rewriting to extraordinary levels, such attacks
are becoming even harder to detect. We conduct a comprehensive investigation of
the role of black-box generative models as a backdoor attack tool, highlighting
the importance of researching relative defense strategies. In this paper, we
reveal that the proposed generative model-based attack, BGMAttack, could
effectively deceive textual classifiers. Compared with the traditional attack
methods, BGMAttack makes the backdoor trigger less conspicuous by leveraging
state-of-the-art generative models. Our extensive evaluation of attack
effectiveness across five datasets, complemented by three distinct human
cognition assessments, reveals that Figure 4 achieves comparable attack
performance while maintaining superior stealthiness relative to baseline
methods.
( 2
min )
We present a new approach, the Topograph, which reconstructs underlying
physics processes, including the intermediary particles, by leveraging
underlying priors from the nature of particle physics decays and the
flexibility of message passing graph neural networks. The Topograph not only
solves the combinatoric assignment of observed final state objects, associating
them to their original mother particles, but directly predicts the properties
of intermediate particles in hard scatter processes and their subsequent
decays. In comparison to standard combinatoric approaches or modern approaches
using graph neural networks, which scale exponentially or quadratically, the
complexity of Topographs scales linearly with the number of reconstructed
objects.
We apply Topographs to top quark pair production in the all hadronic decay
channel, where we outperform the standard approach and match the performance of
the state-of-the-art machine learning technique.
( 2
min )
With recent advancements in computer vision as well as machine learning (ML),
video-based at-home exercise evaluation systems have become a popular topic of
current research. However, performance depends heavily on the amount of
available training data. Since labeled datasets specific to exercising are
rare, we propose a method that makes use of the abundance of fitness videos
available online. Specifically, we utilize the advantage that videos often not
only show the exercises, but also provide language as an additional source of
information. With push-ups as an example, we show that through the analysis of
subtitle data using natural language processing (NLP), it is possible to create
a labeled (irrelevant, relevant correct, relevant incorrect) dataset containing
relevant information for pose analysis. In particular, we show that irrelevant
clips ($n=332$) have significantly different joint visibility values compared
to relevant clips ($n=298$). Inspecting cluster centroids also show different
poses for the different classes.
( 2
min )
The recent advent of play-to-earn (P2E) systems in massively multiplayer
online role-playing games (MMORPGs) has made in-game goods interchangeable with
real-world values more than ever before. The goods in the P2E MMORPGs can be
directly exchanged with cryptocurrencies such as Bitcoin, Ethereum, or Klaytn
via blockchain networks. Unlike traditional in-game goods, once they had been
written to the blockchains, P2E goods cannot be restored by the game operation
teams even with chargeback fraud such as payment fraud, cancellation, or
refund. To tackle the problem, we propose a novel chargeback fraud prediction
method, PU GNN, which leverages graph attention networks with PU loss to
capture both the players' in-game behavior with P2E token transaction patterns.
With the adoption of modified GraphSMOTE, the proposed model handles the
imbalanced distribution of labels in chargeback fraud datasets. The conducted
experiments on three real-world P2E MMORPG datasets demonstrate that PU GNN
achieves superior performances over previously suggested methods.
( 2
min )
Wind energy is a widely distributed, recyclable and environmentally friendly
energy source that plays an important role in mitigating global warming and
energy shortages. Wind energy's uncertainty and fluctuating nature makes grid
integration of large-scale wind energy systems challenging. Medium-term wind
power forecasts can provide an essential basis for energy dispatch, so accurate
wind power forecasts are essential. Much research has yielded excellent results
in recent years. However, many of them require additional experimentation and
analysis when applied to other data. In this paper, we propose a novel
short-term forecasting framework by tree-structured parzen estimator (TPE) and
decomposition algorithms. This framework defines the TPE-VMD-TFT method for
24-h and 48-h ahead wind power forecasting based on variational mode
decomposition (VMD) and time fusion transformer (TFT). In the Engie wind
dataset from the electricity company in France, the results show that the
proposed method significantly improves the prediction accuracy. In addition,
the proposed framework can be used to other decomposition algorithms and
require little manual work in model training.
( 2
min )
Recent work has shown that simple linear models can outperform several
Transformer based approaches in long term time-series forecasting. Motivated by
this, we propose a Multi-layer Perceptron (MLP) based encoder-decoder model,
Time-series Dense Encoder (TiDE), for long-term time-series forecasting that
enjoys the simplicity and speed of linear models while also being able to
handle covariates and non-linear dependencies. Theoretically, we prove that the
simplest linear analogue of our model can achieve near optimal error rate for
linear dynamical systems (LDS) under some assumptions. Empirically, we show
that our method can match or outperform prior approaches on popular long-term
time-series forecasting benchmarks while being 5-10x faster than the best
Transformer based model.
( 2
min )
Computer vision methods have shown to be effective in classifying garbage
into recycling categories for waste processing, existing methods are costly,
imprecise, and unclear. To tackle this issue, we introduce MWaste, a mobile
application that uses computer vision and deep learning techniques to classify
waste materials as trash, plastic, paper, metal, glass or cardboard. Its
effectiveness was tested on various neural network architectures and real-world
images, achieving an average precision of 92\% on the test set. This app can
help combat climate change by enabling efficient waste processing and reducing
the generation of greenhouse gases caused by incorrect waste disposal.
( 2
min )
Detecting an abrupt distributional shift of the data stream, known as
change-point detection, is a fundamental problem in statistics and signal
processing. We present a new approach for online change-point detection by
training neural networks (NN), and sequentially cumulating the detection
statistics by evaluating the trained discriminating function on test samples by
a CUSUM recursion. The idea is based on the observation that training neural
networks through logistic loss may lead to the log-likelihood function. We
demonstrated the good performance of NN-CUSUM in the detection of
high-dimensional data using both synthetic and real-world data.
( 2
min )
Transfer learning (TL) approaches have shown promising results when handling
tasks with limited training data. However, considerable memory and
computational resources are often required for fine-tuning pre-trained neural
networks with target domain data. In this work, we introduce a novel method for
leveraging pre-trained models for low-resource (music) classification based on
the concept of Neural Model Reprogramming (NMR). NMR aims at re-purposing a
pre-trained model from a source domain to a target domain by modifying the
input of a frozen pre-trained model. In addition to the known,
input-independent, reprogramming method, we propose an advanced reprogramming
paradigm: Input-dependent NMR, to increase adaptability to complex input data
such as musical audio. Experimental results suggest that a neural model
pre-trained on large-scale datasets can successfully perform music genre
classification by using this reprogramming method. The two proposed
Input-dependent NMR TL methods outperform fine-tuning-based TL methods on a
small genre classification dataset.
( 2
min )
Feedback from active galactic nuclei (AGN) and supernovae can affect
measurements of integrated SZ flux of halos ($Y_\mathrm{SZ}$) from CMB surveys,
and cause its relation with the halo mass ($Y_\mathrm{SZ}-M$) to deviate from
the self-similar power-law prediction of the virial theorem. We perform a
comprehensive study of such deviations using CAMELS, a suite of hydrodynamic
simulations with extensive variations in feedback prescriptions. We use a
combination of two machine learning tools (random forest and symbolic
regression) to search for analogues of the $Y-M$ relation which are more robust
to feedback processes for low masses ($M\lesssim 10^{14}\, h^{-1} \, M_\odot$);
we find that simply replacing $Y\rightarrow Y(1+M_*/M_\mathrm{gas})$ in the
relation makes it remarkably self-similar. This could serve as a robust
multiwavelength mass proxy for low-mass clusters and galaxy groups. Our
methodology can also be generally useful to improve the domain of validity of
other astrophysical scaling relations.
We also forecast that measurements of the $Y-M$ relation could provide
percent-level constraints on certain combinations of feedback parameters and/or
rule out a major part of the parameter space of supernova and AGN feedback
models used in current state-of-the-art hydrodynamic simulations. Our results
can be useful for using upcoming SZ surveys (e.g., SO, CMB-S4) and galaxy
surveys (e.g., DESI and Rubin) to constrain the nature of baryonic feedback.
Finally, we find that the an alternative relation, $Y-M_*$, provides
complementary information on feedback than $Y-M$
( 3
min )
The Hopfield model is a paradigmatic model of neural networks that has been
analyzed for many decades in the statistical physics, neuroscience, and machine
learning communities. Inspired by the manifold hypothesis in machine learning,
we propose and investigate a generalization of the standard setting that we
name Random-Features Hopfield Model. Here $P$ binary patterns of length $N$ are
generated by applying to Gaussian vectors sampled in a latent space of
dimension $D$ a random projection followed by a non-linearity. Using the
replica method from statistical physics, we derive the phase diagram of the
model in the limit $P,N,D\to\infty$ with fixed ratios $\alpha=P/N$ and
$\alpha_D=D/N$. Besides the usual retrieval phase, where the patterns can be
dynamically recovered from some initial corruption, we uncover a new phase
where the features characterizing the projection can be recovered instead. We
call this phenomena the learning phase transition, as the features are not
explicitly given to the model but rather are inferred from the patterns in an
unsupervised fashion.
( 2
min )
Recently, quantum classifiers have been found to be vulnerable to adversarial
attacks, in which quantum classifiers are deceived by imperceptible noises,
leading to misclassification. In this paper, we propose the first theoretical
study demonstrating that adding quantum random rotation noise can improve
robustness in quantum classifiers against adversarial attacks. We link the
definition of differential privacy and show that the quantum classifier trained
with the natural presence of additive noise is differentially private. Finally,
we derive a certified robustness bound to enable quantum classifiers to defend
against adversarial examples, supported by experimental results simulated with
noises from IBM's 7-qubits device.
( 2
min )
Semantic segmentation models classifying hyperspectral images (HSI) are
vulnerable to adversarial examples. Traditional approaches to adversarial
robustness focus on training or retraining a single network on attacked data,
however, in the presence of multiple attacks these approaches decrease in
performance compared to networks trained individually on each attack. To combat
this issue we propose an Adversarial Discriminator Ensemble Network (ADE-Net)
which focuses on attack type detection and adversarial robustness under a
unified model to preserve per data-type weight optimally while robustifiying
the overall network. In the proposed method, a discriminator network is used to
separate data by attack type into their specific attack-expert ensemble
network.
( 2
min )
We study reward poisoning attacks on online deep reinforcement learning
(DRL), where the attacker is oblivious to the learning algorithm used by the
agent and the dynamics of the environment. We demonstrate the intrinsic
vulnerability of state-of-the-art DRL algorithms by designing a general,
black-box reward poisoning framework called adversarial MDP attacks. We
instantiate our framework to construct two new attacks which only corrupt the
rewards for a small fraction of the total training timesteps and make the agent
learn a low-performing policy. We provide a theoretical analysis of the
efficiency of our attack and perform an extensive empirical evaluation. Our
results show that our attacks efficiently poison agents learning in several
popular classical control and MuJoCo environments with a variety of
state-of-the-art DRL algorithms, such as DQN, PPO, SAC, etc.
( 2
min )
Voxel-based 3D object classification has been thoroughly studied in recent
years. Most previous methods convert the classic 2D convolution into a 3D form
that will be further applied to objects with binary voxel representation for
classification. However, the binary voxel representation is not very effective
for 3D convolution in many cases. In this paper, we propose a hybrid cascade
architecture for voxel-based 3D object classification. It consists of three
stages composed of fully connected and convolutional layers, dealing with easy,
moderate, and hard 3D models respectively. Both accuracy and speed can be
balanced in our proposed method. By giving each voxel a signed distance value,
an obvious gain regarding the accuracy can be observed. Besides, the mean
inference time can be speeded up hugely compared with the state-of-the-art
point cloud and voxel based methods.
( 2
min )
SplitFed Learning, a combination of Federated and Split Learning (FL and SL),
is one of the most recent developments in the decentralized machine learning
domain. In SplitFed learning, a model is trained by clients and a server
collaboratively. For image segmentation, labels are created at each client
independently and, therefore, are subject to clients' bias, inaccuracies, and
inconsistencies. In this paper, we propose a data quality-based adaptive
averaging strategy for SplitFed learning, called QA-SplitFed, to cope with the
variation of annotated ground truth (GT) quality over multiple clients. The
proposed method is compared against five state-of-the-art model averaging
methods on the task of learning human embryo image segmentation. Our
experiments show that all five baseline methods fail to maintain accuracy as
the number of corrupted clients increases. QA-SplitFed, however, copes
effectively with corruption as long as there is at least one uncorrupted
client.
( 2
min )
A stochastic-gradient-based interior-point algorithm for minimizing a
continuously differentiable objective function (that may be nonconvex) subject
to bound constraints is presented, analyzed, and demonstrated through
experimental results. The algorithm is unique from other interior-point methods
for solving smooth (nonconvex) optimization problems since the search
directions are computed using stochastic gradient estimates. It is also unique
in its use of inner neighborhoods of the feasible region -- defined by a
positive and vanishing neighborhood-parameter sequence -- in which the iterates
are forced to remain. It is shown that with a careful balance between the
barrier, step-size, and neighborhood sequences, the proposed algorithm
satisfies convergence guarantees in both deterministic and stochastic settings.
The results of numerical experiments show that in both settings the algorithm
can outperform a projected-(stochastic)-gradient method.
( 2
min )
Privacy-preserving machine learning solutions have recently gained
significant attention. One promising research trend is using Homomorphic
Encryption (HE), a method for performing computation over encrypted data. One
major challenge in this approach is training HE-friendly, encrypted or
unencrypted, deep CNNs with decent accuracy. We propose a novel training method
for HE-friendly models, and demonstrate it on fundamental and modern CNNs, such
as ResNet and ConvNeXt. After training, we evaluate our models by running
encrypted samples using HELayers SDK and proving that they yield the desired
results. When running on a GPU over the ImageNet dataset, our ResNet-18/50/101
implementations take only 7, 31 and 57 minutes, respectively, which shows that
this solution is practical. Furthermore, we present several insights on
handling the activation functions and skip-connections under HE. Finally, we
demonstrate in an unprecedented way how to perform secure zero-shot prediction
using a CLIP model that we adapted to be HE-friendly.
( 2
min )
The ACM Multimedia 2023 Computational Paralinguistics Challenge addresses two
different problems for the first time in a research competition under
well-defined conditions: In the Emotion Share Sub-Challenge, a regression on
speech has to be made; and in the Requests Sub-Challenges, requests and
complaints need to be detected. We describe the Sub-Challenges, baseline
feature extraction, and classifiers based on the usual ComPaRE features, the
auDeep toolkit, and deep feature extraction from pre-trained CNNs using the
DeepSpectRum toolkit; in addition, wav2vec2 models are used.
( 2
min )
Patient-independent detection of epileptic activities based on visual
spectral representation of continuous EEG (cEEG) has been widely used for
diagnosing epilepsy. However, precise detection remains a considerable
challenge due to subtle variabilities across subjects, channels and time
points. Thus, capturing fine-grained, discriminative features of EEG patterns,
which is associated with high-frequency textural information, is yet to be
resolved. In this work, we propose Scattering Transformer (ScatterFormer), an
invariant scattering transform-based hierarchical Transformer that specifically
pays attention to subtle features. In particular, the disentangled
frequency-aware attention (FAA) enables the Transformer to capture clinically
informative high-frequency components, offering a novel clinical explainability
based on visual encoding of multichannel EEG signals. Evaluations on two
distinct tasks of epileptiform detection demonstrate the effectiveness our
method. Our proposed model achieves median AUCROC and accuracy of 98.14%,
96.39% in patients with Rolandic epilepsy. On a neonatal seizure detection
benchmark, it outperforms the state-of-the-art by 9% in terms of average
AUCROC.
( 2
min )
This paper targets the perceptual task of separating the different
interacting voices, i.e., monophonic melodic streams, in a polyphonic musical
piece. We target symbolic music, where notes are explicitly encoded, and model
this task as a Multi-Trajectory Tracking (MTT) problem from discrete
observations, i.e., notes in a pitch-time space. Our approach builds a graph
from a musical piece, by creating one node for every note, and separates the
melodic trajectories by predicting a link between two notes if they are
consecutive in the same voice/stream. This kind of local, greedy prediction is
made possible by node embeddings created by a heterogeneous graph neural
network that can capture inter- and intra-trajectory information. Furthermore,
we propose a new regularization loss that encourages the output to respect the
MTT premise of at most one incoming and one outgoing link for every node,
favouring monophonic (voice) trajectories; this loss function might also be
useful in other general MTT scenarios. Our approach does not use
domain-specific heuristics, is scalable to longer sequences and a higher number
of voices, and can handle complex cases such as voice inversions and overlaps.
We reach new state-of-the-art results for the voice separation task in
classical music of different styles.
( 2
min )
Graph Neural Networks (GNNs) are a form of deep learning that enable a wide
range of machine learning applications on graph-structured data. The learning
of GNNs, however, is known to pose challenges for memory-constrained devices
such as GPUs. In this paper, we study exact compression as a way to reduce the
memory requirements of learning GNNs on large graphs. In particular, we adopt a
formal approach to compression and propose a methodology that transforms GNN
learning problems into provably equivalent compressed GNN learning problems. In
a preliminary experimental evaluation, we give insights into the compression
ratios that can be obtained on real-world graphs and apply our methodology to
an existing GNN benchmark.
( 2
min )
Kernelized Stein discrepancy (KSD) is a score-based discrepancy widely used
in goodness-of-fit tests. It can be applied even when the target distribution
has an unknown normalising factor, such as in Bayesian analysis. We show
theoretically and empirically that the KSD test can suffer from low power when
the target and the alternative distribution have the same well-separated modes
but differ in mixing proportions. We propose to perturb the observed sample via
Markov transition kernels, with respect to which the target distribution is
invariant. This allows us to then employ the KSD test on the perturbed sample.
We provide numerical evidence that with suitably chosen kernels the proposed
approach can lead to a substantially higher power than the KSD test.
( 2
min )
Experimental data is often comprised of variables measured independently, at
different sampling rates (non-uniform ${\Delta}$t between successive
measurements); and at a specific time point only a subset of all variables may
be sampled. Approaches to identifying dynamical systems from such data
typically use interpolation, imputation or subsampling to reorganize or modify
the training data $\textit{prior}$ to learning. Partial physical knowledge may
also be available $\textit{a priori}$ (accurately or approximately), and
data-driven techniques can complement this knowledge. Here we exploit neural
network architectures based on numerical integration methods and $\textit{a
priori}$ physical knowledge to identify the right-hand side of the underlying
governing differential equations. Iterates of such neural-network models allow
for learning from data sampled at arbitrary time points $\textit{without}$ data
modification. Importantly, we integrate the network with available partial
physical knowledge in "physics informed gray-boxes"; this enables learning
unknown kinetic rates or microbial growth functions while simultaneously
estimating experimental parameters.
( 2
min )
Recent work has shown that simple linear models can outperform several
Transformer based approaches in long term time-series forecasting. Motivated by
this, we propose a Multi-layer Perceptron (MLP) based encoder-decoder model,
Time-series Dense Encoder (TiDE), for long-term time-series forecasting that
enjoys the simplicity and speed of linear models while also being able to
handle covariates and non-linear dependencies. Theoretically, we prove that the
simplest linear analogue of our model can achieve near optimal error rate for
linear dynamical systems (LDS) under some assumptions. Empirically, we show
that our method can match or outperform prior approaches on popular long-term
time-series forecasting benchmarks while being 5-10x faster than the best
Transformer based model.
( 2
min )
Kernelized Stein discrepancy (KSD) is a score-based discrepancy widely used
in goodness-of-fit tests. It can be applied even when the target distribution
has an unknown normalising factor, such as in Bayesian analysis. We show
theoretically and empirically that the KSD test can suffer from low power when
the target and the alternative distribution have the same well-separated modes
but differ in mixing proportions. We propose to perturb the observed sample via
Markov transition kernels, with respect to which the target distribution is
invariant. This allows us to then employ the KSD test on the perturbed sample.
We provide numerical evidence that with suitably chosen kernels the proposed
approach can lead to a substantially higher power than the KSD test.
( 2
min )
We provide an exact expressions for the 1-Wasserstein distance between
independent location-scale distributions. The expressions are represented using
location and scale parameters and special functions such as the standard
Gaussian CDF or the Gamma function. Specifically, we find that the
1-Wasserstein distance between independent univariate location-scale
distributions is equivalent to the mean of a folded distribution within the
same family whose underlying location and scale are equal to the difference of
the locations and scales of the original distributions. A new linear upper
bound on the 1-Wasserstein distance is presented and the asymptotic bounds of
the 1-Wasserstein distance are detailed in the Gaussian case. The effect of
differential privacy using the Laplace and Gaussian mechanisms on the
1-Wasserstein distance is studied using the closed-form expressions and bounds.
( 2
min )
Sparse principal component analysis (SPCA) is widely used for dimensionality
reduction and feature extraction in high-dimensional data analysis. Despite
many methodological and theoretical developments in the past two decades, the
theoretical guarantees of the popular SPCA algorithm proposed by Zou, Hastie &
Tibshirani (2006) are still unknown. This paper aims to address this critical
gap. We first revisit the SPCA algorithm of Zou et al. (2006) and present our
implementation. We also study a computationally more efficient variant of the
SPCA algorithm in Zou et al. (2006) that can be considered as the limiting case
of SPCA. We provide the guarantees of convergence to a stationary point for
both algorithms and prove that, under a sparse spiked covariance model, both
algorithms can recover the principal subspace consistently under mild
regularity conditions. We show that their estimation error bounds match the
best available bounds of existing works or the minimax rates up to some
logarithmic factors. Moreover, we demonstrate the competitive numerical
performance of both algorithms in numerical studies.
( 2
min )
Message Passing Neural Networks (MPNNs) are instances of Graph Neural
Networks that leverage the graph to send messages over the edges. This
inductive bias leads to a phenomenon known as over-squashing, where a node
feature is insensitive to information contained at distant nodes. Despite
recent methods introduced to mitigate this issue, an understanding of the
causes for over-squashing and of possible solutions are lacking. In this
theoretical work, we prove that: (i) Neural network width can mitigate
over-squashing, but at the cost of making the whole network more sensitive;
(ii) Conversely, depth cannot help mitigate over-squashing: increasing the
number of layers leads to over-squashing being dominated by vanishing
gradients; (iii) The graph topology plays the greatest role, since
over-squashing occurs between nodes at high commute (access) time. Our analysis
provides a unified framework to study different recent methods introduced to
cope with over-squashing and serves as a justification for a class of methods
that fall under `graph rewiring'.
( 2
min )
Recovering the latent factors of variation of high dimensional data has so
far focused on simple synthetic settings. Mostly building on unsupervised and
weakly-supervised objectives, prior work missed out on the positive
implications for representation learning on real world data. In this work, we
propose to leverage knowledge extracted from a diversified set of supervised
tasks to learn a common disentangled representation. Assuming each supervised
task only depends on an unknown subset of the factors of variation, we
disentangle the feature space of a supervised multi-task model, with features
activating sparsely across different tasks and information being shared as
appropriate. Importantly, we never directly observe the factors of variations
but establish that access to multiple tasks is sufficient for identifiability
under sufficiency and minimality assumptions. We validate our approach on six
real world distribution shift benchmarks, and different data modalities
(images, text), demonstrating how disentangled representations can be
transferred to real settings.
( 2
min )
We study variance-dependent regret bounds for Markov decision processes
(MDPs). Algorithms with variance-dependent regret guarantees can automatically
exploit environments with low variance (e.g., enjoying constant regret on
deterministic MDPs). The existing algorithms are either variance-independent or
suboptimal. We first propose two new environment norms to characterize the
fine-grained variance properties of the environment. For model-based methods,
we design a variant of the MVP algorithm (Zhang et al., 2021a) and use new
analysis techniques show to this algorithm enjoys variance-dependent bounds
with respect to our proposed norms. In particular, this bound is simultaneously
minimax optimal for both stochastic and deterministic MDPs, the first result of
its kind. We further initiate the study on model-free algorithms with
variance-dependent regret bounds by designing a reference-function-based
algorithm with a novel capped-doubling reference update schedule. Lastly, we
also provide lower bounds to complement our upper bounds.
( 2
min )
A spiking neural network (SNN) equalizer with a decision feedback structure
is applied to an IM/DD link with various parameters. The SNN outperforms linear
and artificial neural network (ANN) based equalizers.
( 2
min )
The goal of this paper is to learn more about how idiomatic information is
structurally encoded in embeddings, using a structural probing method. We
repurpose an existing English verbal multi-word expression (MWE) dataset to
suit the probing framework and perform a comparative probing study of static
(GloVe) and contextual (BERT) embeddings. Our experiments indicate that both
encode some idiomatic information to varying degrees, but yield conflicting
evidence as to whether idiomaticity is encoded in the vector norm, leaving this
an open question. We also identify some limitations of the used dataset and
highlight important directions for future work in improving its suitability for
a probing analysis.
( 2
min )
Experimental data is often comprised of variables measured independently, at
different sampling rates (non-uniform ${\Delta}$t between successive
measurements); and at a specific time point only a subset of all variables may
be sampled. Approaches to identifying dynamical systems from such data
typically use interpolation, imputation or subsampling to reorganize or modify
the training data $\textit{prior}$ to learning. Partial physical knowledge may
also be available $\textit{a priori}$ (accurately or approximately), and
data-driven techniques can complement this knowledge. Here we exploit neural
network architectures based on numerical integration methods and $\textit{a
priori}$ physical knowledge to identify the right-hand side of the underlying
governing differential equations. Iterates of such neural-network models allow
for learning from data sampled at arbitrary time points $\textit{without}$ data
modification. Importantly, we integrate the network with available partial
physical knowledge in "physics informed gray-boxes"; this enables learning
unknown kinetic rates or microbial growth functions while simultaneously
estimating experimental parameters.
( 2
min )
Annealed Importance Sampling (AIS) moves particles along a Markov chain from
a tractable initial distribution to an intractable target distribution. The
recently proposed Differentiable AIS (DAIS) (Geffner and Domke, 2021; Zhang et
al., 2021) enables efficient optimization of the transition kernels of AIS and
of the distributions. However, we observe a low effective sample size in DAIS,
indicating degenerate distributions. We thus propose to extend DAIS by a
resampling step inspired by Sequential Monte Carlo. Surprisingly, we find
empirically-and can explain theoretically-that it is not necessary to
differentiate through the resampling step which avoids gradient variance issues
observed in similar approaches for Particle Filters (Maddison et al., 2017;
Naesseth et al., 2018; Le et al., 2018).
( 2
min )
Organizations are increasingly adopting machine learning (ML) for personnel
assessment. However, concerns exist about fairness in designing and
implementing ML assessments. Supervised ML models are trained to model patterns
in data, meaning ML models tend to yield predictions that reflect subgroup
differences in applicant attributes in the training data, regardless of the
underlying cause of subgroup differences. In this study, we systematically
under- and oversampled minority (Black and Hispanic) applicants to manipulate
adverse impact ratios in training data and investigated how training data
adverse impact ratios affect ML model adverse impact and accuracy. We used
self-reports and interview transcripts from job applicants (N = 2,501) to train
9,702 ML models to predict screening decisions. We found that training data
adverse impact related linearly to ML model adverse impact. However, removing
adverse impact from training data only slightly reduced ML model adverse impact
and tended to negatively affect ML model accuracy. We observed consistent
effects across self-reports and interview transcripts, whether oversampling
real (i.e., bootstrapping) or synthetic observations. As our study relied on
limited predictor sets from one organization, the observed effects on adverse
impact may be attenuated among more accurate ML models.
( 2
min )
This tutorial survey provides an overview of recent non-asymptotic advances
in statistical learning theory as relevant to control and system
identification. While there has been substantial progress across all areas of
control, the theory is most well-developed when it comes to linear system
identification and learning for the linear quadratic regulator, which are the
focus of this manuscript. From a theoretical perspective, much of the labor
underlying these advances has been in adapting tools from modern
high-dimensional statistics and learning theory. While highly relevant to
control theorists interested in integrating tools from machine learning, the
foundational material has not always been easily accessible. To remedy this, we
provide a self-contained presentation of the relevant material, outlining all
the key ideas and the technical machinery that underpin recent results. We also
present a number of open problems and future directions.
( 2
min )
Conditional Average Treatment Effects (CATE) estimation is one of the main
challenges in causal inference with observational data. In addition to Machine
Learning based-models, nonparametric estimators called meta-learners have been
developed to estimate the CATE with the main advantage of not restraining the
estimation to a specific supervised learning method. This task becomes,
however, more complicated when the treatment is not binary as some limitations
of the naive extensions emerge. This paper looks into meta-learners for
estimating the heterogeneous effects of multi-valued treatments. We consider
different meta-learners, and we carry out a theoretical analysis of their error
upper bounds as functions of important parameters such as the number of
treatment levels, showing that the naive extensions do not always provide
satisfactory results. We introduce and discuss meta-learners that perform well
as the number of treatments increases. We empirically confirm the strengths and
weaknesses of those methods with synthetic and semi-synthetic datasets.
( 2
min )
A crucial challenge in reinforcement learning is to reduce the number of
interactions with the environment that an agent requires to master a given
task. Transfer learning proposes to address this issue by re-using knowledge
from previously learned tasks. However, determining which source task qualifies
as the most appropriate for knowledge extraction, as well as the choice
regarding which algorithm components to transfer, represent severe obstacles to
its application in reinforcement learning. The goal of this paper is to address
these issues with modular multi-source transfer learning techniques. The
proposed techniques automatically learn how to extract useful information from
source tasks, regardless of the difference in state-action space and reward
function. We support our claims with extensive and challenging cross-domain
experiments for visual control.
( 2
min )
We propose the first online quantum algorithm for zero-sum games with $\tilde
O(1)$ regret under the game setting. Moreover, our quantum algorithm computes
an $\varepsilon$-approximate Nash equilibrium of an $m \times n$ matrix
zero-sum game in quantum time $\tilde O(\sqrt{m+n}/\varepsilon^{2.5})$,
yielding a quadratic improvement over classical algorithms in terms of $m, n$.
Our algorithm uses standard quantum inputs and generates classical outputs with
succinct descriptions, facilitating end-to-end applications. As an application,
we obtain a fast quantum linear programming solver. Technically, our online
quantum algorithm "quantizes" classical algorithms based on the optimistic
multiplicative weight update method. At the heart of our algorithm is a fast
quantum multi-sampling procedure for the Gibbs sampling problem, which may be
of independent interest.
( 2
min )
We analyze the generalization ability of joint-training meta learning
algorithms via the Gibbs algorithm. Our exact characterization of the expected
meta generalization error for the meta Gibbs algorithm is based on symmetrized
KL information, which measures the dependence between all meta-training
datasets and the output parameters, including task-specific and meta
parameters. Additionally, we derive an exact characterization of the meta
generalization error for the super-task Gibbs algorithm, in terms of
conditional symmetrized KL information within the super-sample and super-task
framework introduced in Steinke and Zakynthinou (2020) and Hellstrom and Durisi
(2022) respectively. Our results also enable us to provide novel
distribution-free generalization error upper bounds for these Gibbs algorithms
applicable to meta learning.
( 2
min )
Many techniques in machine learning attempt explicitly or implicitly to infer
a low-dimensional manifold structure of an underlying physical phenomenon from
measurements without an explicit model of the phenomenon or the measurement
apparatus. This paper presents a cautionary tale regarding the discrepancy
between the geometry of measurements and the geometry of the underlying
phenomenon in a benign setting. The deformation in the metric illustrated in
this paper is mathematically straightforward and unavoidable in the general
case, and it is only one of several similar effects. While this is not always
problematic, we provide an example of an arguably standard and harmless data
processing procedure where this effect leads to an incorrect answer to a
seemingly simple question. Although we focus on manifold learning, these issues
apply broadly to dimensionality reduction and unsupervised learning.
( 2
min )
The learnable, linear neural network layers between tensor power spaces of
$\mathbb{R}^{n}$ that are equivariant to the orthogonal group, $O(n)$, the
special orthogonal group, $SO(n)$, and the symplectic group, $Sp(n)$, were
characterised in arXiv:2212.08630. We present an algorithm for multiplying a
vector by any weight matrix for each of these groups, using category theoretic
constructions to implement the procedure. We achieve a significant reduction in
computational cost compared with a naive implementation by making use of
Kronecker product matrices to perform the multiplication. We show that our
approach extends to the symmetric group, $S_n$, recovering the algorithm of
arXiv:2303.06208 in the process.
( 2
min )
Neural network model compression techniques can address the computation issue
of deep neural networks on embedded devices in industrial systems. The
guaranteed output error computation problem for neural network compression with
quantization is addressed in this paper. A merged neural network is built from
a feedforward neural network and its quantized version to produce the exact
output difference between two neural networks. Then, optimization-based methods
and reachability analysis methods are applied to the merged neural network to
compute the guaranteed quantization error. Finally, a numerical example is
proposed to validate the applicability and effectiveness of the proposed
approach.
( 2
min )
We introduce a new computational framework for estimating parameters in
generalized generalized linear models (GGLM), a class of models that extends
the popular generalized linear models (GLM) to account for dependencies among
observations in spatio-temporal data. The proposed approach uses a monotone
operator-based variational inequality method to overcome non-convexity in
parameter estimation and provide guarantees for parameter recovery. The results
can be applied to GLM and GGLM, focusing on spatio-temporal models. We also
present online instance-based bounds using martingale concentrations
inequalities. Finally, we demonstrate the performance of the algorithm using
numerical simulations and a real data example for wildfire incidents.
( 2
min )
Gradient-boosted decision trees (GBDT) are widely used and highly effective
machine learning approach for tabular data modeling. However, their complex
structure may lead to low robustness against small covariate perturbation in
unseen data. In this study, we apply one-hot encoding to convert a GBDT model
into a linear framework, through encoding of each tree leaf to one dummy
variable. This allows for the use of linear regression techniques, plus a novel
risk decomposition for assessing the robustness of a GBDT model against
covariate perturbations. We propose to enhance the robustness of GBDT models by
refitting their linear regression forms with $L_1$ or $L_2$ regularization.
Theoretical results are obtained about the effect of regularization on the
model performance and robustness. It is demonstrated through numerical
experiments that the proposed regularization approach can enhance the
robustness of the one-hot-encoded GBDT models.
( 2
min )
In this paper, a computationally efficient data-driven hybrid automaton model
is proposed to capture unknown complex dynamical system behaviors using
multiple neural networks. The sampled data of the system is divided by valid
partitions into groups corresponding to their topologies and based on which,
transition guards are defined. Then, a collection of small-scale neural
networks that are computationally efficient are trained as the local dynamical
description for their corresponding topologies. After modeling the system with
a neural-network-based hybrid automaton, the set-valued reachability analysis
with low computation cost is provided based on interval analysis and a split
and combined process. At last, a numerical example of the limit cycle is
presented to illustrate that the developed models can significantly reduce the
computational cost in reachable set computation without sacrificing any
modeling precision.
( 2
min )
Federated learning (FL) is an emerging technique that trains massive and
geographically distributed edge data while maintaining privacy. However, FL has
inherent challenges in terms of fairness and computational efficiency due to
the rising heterogeneity of edges, and thus usually results in sub-optimal
performance in recent state-of-the-art (SOTA) solutions. In this paper, we
propose a Customized Federated Learning (CFL) system to eliminate FL
heterogeneity from multiple dimensions. Specifically, CFL tailors personalized
models from the specially designed global model for each client jointly guided
by an online trained model-search helper and a novel aggregation algorithm.
Extensive experiments demonstrate that CFL has full-stack advantages for both
FL training and edge reasoning and significantly improves the SOTA performance
w.r.t. model accuracy (up to 7.2% in the non-heterogeneous environment and up
to 21.8% in the heterogeneous environment), efficiency, and FL fairness.
( 2
min )
Semantic knowledge of part-part and part-whole relationships in assemblies is
useful for a variety of tasks from searching design repositories to the
construction of engineering knowledge bases. In this work we propose that the
natural language names designers use in Computer Aided Design (CAD) software
are a valuable source of such knowledge, and that Large Language Models (LLMs)
contain useful domain-specific information for working with this data as well
as other CAD and engineering-related tasks.
In particular we extract and clean a large corpus of natural language part,
feature and document names and use this to quantitatively demonstrate that a
pre-trained language model can outperform numerous benchmarks on three
self-supervised tasks, without ever having seen this data before. Moreover, we
show that fine-tuning on the text data corpus further boosts the performance on
all tasks, thus demonstrating the value of the text data which until now has
been largely ignored. We also identify key limitations to using LLMs with text
data alone, and our findings provide a strong motivation for further work into
multi-modal text-geometry models.
To aid and encourage further work in this area we make all our data and code
publicly available.
( 2
min )
In this paper we present the first version of ganX -- generate artificially
new XRF, a Python library to generate X-ray fluorescence Macro maps (MA-XRF)
from a coloured RGB image. To do that, a Monte Carlo method is used, where each
MA-XRF pixel signal is sampled out of an XRF signal probability function. Such
probability function is computed using a database of couples (pigment
characteristic XRF signal, RGB), by a weighted sum of such pigment XRF signal
by proximity of the image RGB to the pigment characteristic RGB. The library is
released to PyPi and the code is available open source on GitHub.
( 2
min )
Distributed learning paradigms, such as federated or decentralized learning,
allow a collection of agents to solve global learning and optimization problems
through limited local interactions. Most such strategies rely on a mixture of
local adaptation and aggregation steps, either among peers or at a central
fusion center. Classically, aggregation in distributed learning is based on
averaging, which is statistically efficient, but susceptible to attacks by even
a small number of malicious agents. This observation has motivated a number of
recent works, which develop robust aggregation schemes by employing robust
variations of the mean. We present a new attack based on sensitivity curve
maximization (SCM), and demonstrate that it is able to disrupt existing robust
aggregation schemes by injecting small, but effective perturbations.
( 2
min )
This paper provides answers to an open problem: given a nonlinear data-driven
dynamical system model, e.g., kernel conditional mean embedding (CME) and
Koopman operator, how can one propagate the ambiguity sets forward for multiple
steps? This problem is the key to solving distributionally robust control and
learning-based control of such learned system models under a data-distribution
shift. Different from previous works that use either static ambiguity sets,
e.g., fixed Wasserstein balls, or dynamic ambiguity sets under known piece-wise
linear (or affine) dynamics, we propose an algorithm that exactly propagates
ambiguity sets through nonlinear data-driven models using the Koopman operator
and CME, via the kernel maximum mean discrepancy geometry. Through both
theoretical and numerical analysis, we show that our kernel ambiguity sets are
the natural geometric structure for the learned data-driven dynamical system
models.
( 2
min )
The success of the Adam optimizer on a wide array of architectures has made
it the default in settings where stochastic gradient descent (SGD) performs
poorly. However, our theoretical understanding of this discrepancy is lagging,
preventing the development of significant improvements on either algorithm.
Recent work advances the hypothesis that Adam and other heuristics like
gradient clipping outperform SGD on language tasks because the distribution of
the error induced by sampling has heavy tails. This suggests that Adam
outperform SGD because it uses a more robust gradient estimate. We evaluate
this hypothesis by varying the batch size, up to the entire dataset, to control
for stochasticity. We present evidence that stochasticity and heavy-tailed
noise are not major factors in the performance gap between SGD and Adam.
Rather, Adam performs better as the batch size increases, while SGD is less
effective at taking advantage of the reduction in noise. This raises the
question as to why Adam outperforms SGD in the full-batch setting. Through
further investigation of simpler variants of SGD, we find that the behavior of
Adam with large batches is similar to sign descent with momentum.
( 2
min )
This paper introduces JaxPruner, an open-source JAX-based pruning and sparse
training library for machine learning research. JaxPruner aims to accelerate
research on sparse neural networks by providing concise implementations of
popular pruning and sparse training algorithms with minimal memory and latency
overhead. Algorithms implemented in JaxPruner use a common API and work
seamlessly with the popular optimization library Optax, which, in turn, enables
easy integration with existing JAX based libraries. We demonstrate this ease of
integration by providing examples in four different codebases: Scenic, t5x,
Dopamine and FedJAX and provide baseline experiments on popular benchmarks.
( 2
min )
In recent years, there has been a surge in effort to formalize notions of
fairness in machine learning. We focus on clustering -- one of the fundamental
tasks in unsupervised machine learning. We propose a new axiom that captures
proportional representation fairness (PRF). We make a case that the concept
achieves the raison d'{\^{e}}tre of several existing concepts in the literature
in an arguably more convincing manner. Our fairness concept is not satisfied by
existing fair clustering algorithms. We design efficient algorithms to achieve
PRF both for unconstrained and discrete clustering problems.
( 2
min )
In documents and graphics, contours are a popular format to describe specific
shapes. For example, in the True Type Font (TTF) file format, contours describe
vector outlines of typeface shapes. Each contour is often defined as a sequence
of points. In this paper, we tackle the contour completion task. In this task,
the input is a contour sequence with missing points, and the output is a
generated completed contour. This task is more difficult than image completion
because, for images, the missing pixels are indicated. Since there is no such
indication in the contour completion task, we must solve the problem of missing
part detection and completion simultaneously. We propose a Transformer-based
method to solve this problem and show the results of the typeface contour
completion.
( 2
min )
This study aims to alleviate the trade-off between utility and privacy in the
task of differentially private clustering. Existing works focus on simple
clustering methods, which show poor clustering performance for non-convex
clusters. By utilizing Morse theory, we hierarchically connect the Gaussian
sub-clusters to fit complex cluster distributions. Because differentially
private sub-clusters are obtained through the existing methods, the proposed
method causes little or no additional privacy loss. We provide a theoretical
background that implies that the proposed method is inductive and can achieve
any desired number of clusters. Experiments on various datasets show that our
framework achieves better clustering performance at the same privacy level,
compared to the existing methods.
( 2
min )
In 1-bit matrix completion, the aim is to estimate an underlying low-rank
matrix from a partial set of binary observations. We propose a novel method for
1-bit matrix completion called MMGN. Our method is based on the
majorization-minimization (MM) principle, which yields a sequence of standard
low-rank matrix completion problems in our setting. We solve each of these
sub-problems by a factorization approach that explicitly enforces the assumed
low-rank structure and then apply a Gauss-Newton method. Our numerical studies
and application to a real-data example illustrate that MMGN outputs comparable
if not more accurate estimates, is often significantly faster, and is less
sensitive to the spikiness of the underlying matrix than existing methods.
( 2
min )
In recent years, product categorisation has been a common issue for
E-commerce companies who have utilised machine learning to categorise their
products automatically. In this study, we propose an ensemble approach, using a
combination of different models to separately predict each product's category,
subcategory, and colour before ultimately combining the resultant predictions
for each product. With the aforementioned approach, we show that an average
F1-score of 0.82 can be achieved using a combination of XGBoost and k-nearest
neighbours to predict said features.
( 2
min )
Innovative Electronic Design Automation (EDA) solutions are important to meet
the design requirements for increasingly complex electronic devices. Verilog, a
hardware description language, is widely used for the design and verification
of digital circuits and is synthesized using specific EDA tools. However,
writing code is a repetitive and time-intensive task. This paper proposes,
primarily, a novel deep learning framework for training a Verilog
autocompletion model and, secondarily, a Verilog dataset of files and snippets
obtained from open-source repositories. The framework involves integrating
models pretrained on general programming language data and finetuning them on a
dataset curated to be similar to a target downstream task. This is validated by
comparing different pretrained models trained on different subsets of the
proposed Verilog dataset using multiple evaluation metrics. These experiments
demonstrate that the proposed framework achieves better BLEU, ROUGE-L, and chrF
scores by 9.5%, 6.7%, and 6.9%, respectively, compared to a model trained from
scratch.
( 2
min )
While the use of the Internet of Things is becoming more and more popular,
many security vulnerabilities are emerging with the large number of devices
being introduced to the market. In this environment, IoT device identification
methods provide a preventive security measure as an important factor in
identifying these devices and detecting the vulnerabilities they suffer from.
In this study, we present a method that identifies devices in the Aalto dataset
using Long short-term memory (LSTM)
( 2
min )
In online forums like Reddit, users share their experiences with medical
conditions and treatments, including making claims, asking questions, and
discussing the effects of treatments on their health. Building systems to
understand this information can effectively monitor the spread of
misinformation and verify user claims. The Task-8 of the 2023 International
Workshop on Semantic Evaluation focused on medical applications, specifically
extracting patient experience- and medical condition-related entities from user
posts on social media. The Reddit Health Online Talk (RedHot) corpus contains
posts from medical condition-related subreddits with annotations characterizing
the patient experience and medical conditions. In Subtask-1, patient experience
is characterized by personal experience, questions, and claims. In Subtask-2,
medical conditions are characterized by population, intervention, and outcome.
For the automatic extraction of patient experiences and medical condition
information, as a part of the challenge, we proposed language-model-based
extraction systems that ranked $3^{rd}$ on both subtasks' leaderboards. In this
work, we describe our approach and, in addition, explore the automatic
extraction of this information using domain-specific language models and the
inclusion of external knowledge.
( 2
min )
This paper assesses the reliability of the RemOve-And-Retrain (ROAR)
protocol, which is used to measure the performance of feature importance
estimates. Our findings from the theoretical background and empirical
experiments indicate that attributions that possess less information about the
decision function can perform better in ROAR benchmarks, conflicting with the
original purpose of ROAR. This phenomenon is also observed in the recently
proposed variant RemOve-And-Debias (ROAD), and we propose a consistent trend of
blurriness bias in ROAR attribution metrics. Our results caution against
uncritical reliance on ROAR metrics.
( 2
min )
Current dialogue research primarily studies pairwise (two-party)
conversations, and does not address the everyday setting where more than two
speakers converse together. In this work, we both collect and evaluate
multi-party conversations to study this more general case. We use the LIGHT
environment to construct grounded conversations, where each participant has an
assigned character to role-play. We thus evaluate the ability of language
models to act as one or more characters in such conversations. Models require
two skills that pairwise-trained models appear to lack: (1) being able to
decide when to talk; (2) producing coherent utterances grounded on multiple
characters. We compare models trained on our new dataset to existing
pairwise-trained dialogue models, as well as large language models with
few-shot prompting. We find that our new dataset, MultiLIGHT, which we will
publicly release, can help bring significant improvements in the group setting.
( 2
min )
To improve the recognition ability of computer-aided breast mass
classification among mammographic images, in this work we explore the
state-of-the-art classification networks to develop an ensemble mechanism.
First, the regions of interest (ROIs) are obtained from the original dataset,
and then three models, i.e., XceptionNet, DenseNet, and EfficientNet, are
trained individually. After training, we ensemble the mechanism by summing the
probabilities outputted from each network which enhances the performance up to
5%. The scheme has been validated on a public dataset and we achieved accuracy,
precision, and recall 88%, 85%, and 76% respectively.
( 2
min )
Gradient-boosted decision trees (GBDT) are widely used and highly effective
machine learning approach for tabular data modeling. However, their complex
structure may lead to low robustness against small covariate perturbation in
unseen data. In this study, we apply one-hot encoding to convert a GBDT model
into a linear framework, through encoding of each tree leaf to one dummy
variable. This allows for the use of linear regression techniques, plus a novel
risk decomposition for assessing the robustness of a GBDT model against
covariate perturbations. We propose to enhance the robustness of GBDT models by
refitting their linear regression forms with $L_1$ or $L_2$ regularization.
Theoretical results are obtained about the effect of regularization on the
model performance and robustness. It is demonstrated through numerical
experiments that the proposed regularization approach can enhance the
robustness of the one-hot-encoded GBDT models.
( 2
min )
Annealed Importance Sampling (AIS) moves particles along a Markov chain from
a tractable initial distribution to an intractable target distribution. The
recently proposed Differentiable AIS (DAIS) (Geffner and Domke, 2021; Zhang et
al., 2021) enables efficient optimization of the transition kernels of AIS and
of the distributions. However, we observe a low effective sample size in DAIS,
indicating degenerate distributions. We thus propose to extend DAIS by a
resampling step inspired by Sequential Monte Carlo. Surprisingly, we find
empirically-and can explain theoretically-that it is not necessary to
differentiate through the resampling step which avoids gradient variance issues
observed in similar approaches for Particle Filters (Maddison et al., 2017;
Naesseth et al., 2018; Le et al., 2018).
( 2
min )
We detail an approach to develop Stein's method for bounding integral metrics
on probability measures defined on a Riemannian manifold $\mathbf M$. Our
approach exploits the relationship between the generator of a diffusion on
$\mathbf M$ with target invariant measure and its characterising Stein
operator. We consider a pair of such diffusions with different starting points,
and through analysis of the distance process between the pair, derive Stein
factors, which bound the solution to the Stein equation and its derivatives.
The Stein factors contain curvature-dependent terms and reduce to those
currently available for $\mathbb R^m$, and moreover imply that the bounds for
$\mathbb R^m$ remain valid when $\mathbf M$ is a flat manifold
( 2
min )
Conditional Average Treatment Effects (CATE) estimation is one of the main
challenges in causal inference with observational data. In addition to Machine
Learning based-models, nonparametric estimators called meta-learners have been
developed to estimate the CATE with the main advantage of not restraining the
estimation to a specific supervised learning method. This task becomes,
however, more complicated when the treatment is not binary as some limitations
of the naive extensions emerge. This paper looks into meta-learners for
estimating the heterogeneous effects of multi-valued treatments. We consider
different meta-learners, and we carry out a theoretical analysis of their error
upper bounds as functions of important parameters such as the number of
treatment levels, showing that the naive extensions do not always provide
satisfactory results. We introduce and discuss meta-learners that perform well
as the number of treatments increases. We empirically confirm the strengths and
weaknesses of those methods with synthetic and semi-synthetic datasets.
( 2
min )
We introduce a new computational framework for estimating parameters in
generalized generalized linear models (GGLM), a class of models that extends
the popular generalized linear models (GLM) to account for dependencies among
observations in spatio-temporal data. The proposed approach uses a monotone
operator-based variational inequality method to overcome non-convexity in
parameter estimation and provide guarantees for parameter recovery. The results
can be applied to GLM and GGLM, focusing on spatio-temporal models. We also
present online instance-based bounds using martingale concentrations
inequalities. Finally, we demonstrate the performance of the algorithm using
numerical simulations and a real data example for wildfire incidents.
( 2
min )
Variational Bayes is a popular method for approximate inference but its
derivation can be cumbersome. To simplify the process, we give a 3-step recipe
to identify the posterior form by explicitly looking for linearity with respect
to expectations of well-known distributions. We can then directly write the
update by simply ``reading-off'' the terms in front of those expectations. The
recipe makes the derivation easier, faster, shorter, and more general.
( 2
min )
In 1-bit matrix completion, the aim is to estimate an underlying low-rank
matrix from a partial set of binary observations. We propose a novel method for
1-bit matrix completion called MMGN. Our method is based on the
majorization-minimization (MM) principle, which yields a sequence of standard
low-rank matrix completion problems in our setting. We solve each of these
sub-problems by a factorization approach that explicitly enforces the assumed
low-rank structure and then apply a Gauss-Newton method. Our numerical studies
and application to a real-data example illustrate that MMGN outputs comparable
if not more accurate estimates, is often significantly faster, and is less
sensitive to the spikiness of the underlying matrix than existing methods.
( 2
min )
This tutorial survey provides an overview of recent non-asymptotic advances
in statistical learning theory as relevant to control and system
identification. While there has been substantial progress across all areas of
control, the theory is most well-developed when it comes to linear system
identification and learning for the linear quadratic regulator, which are the
focus of this manuscript. From a theoretical perspective, much of the labor
underlying these advances has been in adapting tools from modern
high-dimensional statistics and learning theory. While highly relevant to
control theorists interested in integrating tools from machine learning, the
foundational material has not always been easily accessible. To remedy this, we
provide a self-contained presentation of the relevant material, outlining all
the key ideas and the technical machinery that underpin recent results. We also
present a number of open problems and future directions.
( 2
min )
Message Passing Neural Networks (MPNNs) are instances of Graph Neural
Networks that leverage the graph to send messages over the edges. This
inductive bias leads to a phenomenon known as over-squashing, where a node
feature is insensitive to information contained at distant nodes. Despite
recent methods introduced to mitigate this issue, an understanding of the
causes for over-squashing and of possible solutions are lacking. In this
theoretical work, we prove that: (i) Neural network width can mitigate
over-squashing, but at the cost of making the whole network more sensitive;
(ii) Conversely, depth cannot help mitigate over-squashing: increasing the
number of layers leads to over-squashing being dominated by vanishing
gradients; (iii) The graph topology plays the greatest role, since
over-squashing occurs between nodes at high commute (access) time. Our analysis
provides a unified framework to study different recent methods introduced to
cope with over-squashing and serves as a justification for a class of methods
that fall under `graph rewiring'.
( 2
min )
The learnable, linear neural network layers between tensor power spaces of
$\mathbb{R}^{n}$ that are equivariant to the orthogonal group, $O(n)$, the
special orthogonal group, $SO(n)$, and the symplectic group, $Sp(n)$, were
characterised in arXiv:2212.08630. We present an algorithm for multiplying a
vector by any weight matrix for each of these groups, using category theoretic
constructions to implement the procedure. We achieve a significant reduction in
computational cost compared with a naive implementation by making use of
Kronecker product matrices to perform the multiplication. We show that our
approach extends to the symmetric group, $S_n$, recovering the algorithm of
arXiv:2303.06208 in the process.
( 2
min )
We study variance-dependent regret bounds for Markov decision processes
(MDPs). Algorithms with variance-dependent regret guarantees can automatically
exploit environments with low variance (e.g., enjoying constant regret on
deterministic MDPs). The existing algorithms are either variance-independent or
suboptimal. We first propose two new environment norms to characterize the
fine-grained variance properties of the environment. For model-based methods,
we design a variant of the MVP algorithm (Zhang et al., 2021a) and use new
analysis techniques show to this algorithm enjoys variance-dependent bounds
with respect to our proposed norms. In particular, this bound is simultaneously
minimax optimal for both stochastic and deterministic MDPs, the first result of
its kind. We further initiate the study on model-free algorithms with
variance-dependent regret bounds by designing a reference-function-based
algorithm with a novel capped-doubling reference update schedule. Lastly, we
also provide lower bounds to complement our upper bounds.
( 2
min )
Many techniques in machine learning attempt explicitly or implicitly to infer
a low-dimensional manifold structure of an underlying physical phenomenon from
measurements without an explicit model of the phenomenon or the measurement
apparatus. This paper presents a cautionary tale regarding the discrepancy
between the geometry of measurements and the geometry of the underlying
phenomenon in a benign setting. The deformation in the metric illustrated in
this paper is mathematically straightforward and unavoidable in the general
case, and it is only one of several similar effects. While this is not always
problematic, we provide an example of an arguably standard and harmless data
processing procedure where this effect leads to an incorrect answer to a
seemingly simple question. Although we focus on manifold learning, these issues
apply broadly to dimensionality reduction and unsupervised learning.
( 2
min )
Wartella and AI reinvigorate a White Stripes classic, exploring AI’s role in music video creation.
( 6
min )
Source Manufacturers often turn to digitalization strategies to improve their competitiveness, address labor shortages, and boost productivity. These efforts are driven by a desire to stay ahead of the game rather than simply defend against the competition. However, moving to the front foot regarding generated data unlocks waves of innovation — creating fast, bold, competitive,… Read More »3 Major Benefits Data Collection Brings To The Manufacturing Process
The post 3 Major Benefits Data Collection Brings To The Manufacturing Process appeared first on Data Science Central.
( 21
min )
I recently completed teaching my “Big Data MBA: Thinking Like a Data Scientist (TLADS)” class for the spring semester at Iowa State University. I had 17 second-year MBA students, and their diligence, passion, and creativity were evident throughout the semester and especially in the final project presentations. This class had no tests or mid-term exams… Read More »Iowa State University: “Thinking Like a Data Scientist” Lessons Learned
The post Iowa State University: “Thinking Like a Data Scientist” Lessons Learned appeared first on Data Science Central.
( 21
min )
A new method could provide detailed information about internal structures, voids, and cracks, based solely on data about exterior conditions.
( 10
min )
Recent large language models (LLMs) have enabled tremendous progress in natural language understanding. However, they are prone to generating confident but nonsensical explanations, which poses a significant obstacle to establishing trust with users. In this post, we show how to incorporate human feedback on the incorrect reasoning chains for multi-hop reasoning to improve performance on […]
( 10
min )
Deep learning (DL) is a fast-evolving field, and practitioners are constantly innovating DL models and inventing ways to speed them up. Custom operators are one of the mechanisms developers use to push the boundaries of DL innovation by extending the functionality of existing machine learning (ML) frameworks such as PyTorch. In general, an operator describes […]
( 11
min )
Horror descends from the cloud this GFN Thursday with the arrival of publisher Capcom’s iconic Resident Evil series. They’re part of nine new games expanding the GeForce NOW library of over 1,600 titles. RTX 4080 SuperPODs are now live in Miami, Portland, Ore., and Stockholm. Follow along with the server rollout process, and make the Read article >
( 4
min )
We employ unsupervised machine learning to enhance the accuracy of our
recently presented scaling method for wave confinement analysis [1]. We employ
the standard k-means++ algorithm as well as our own model-based algorithm. We
investigate cluster validity indices as a means to find the correct number of
confinement dimensionalities to be used as an input to the clustering
algorithms. Subsequently, we analyze the performance of the two clustering
algorithms when compared to the direct application of the scaling method
without clustering. We find that the clustering approach provides more
physically meaningful results, but may struggle with identifying the correct
set of confinement dimensionalities. We conclude that the most accurate outcome
is obtained by first applying the direct scaling to find the correct set of
confinement dimensionalities and subsequently employing clustering to refine
the results. Moreover, our model-based algorithm outperforms the standard
k-means++ clustering.
( 2
min )
We identify and explore connections between the recent literature on
multi-group fairness for prediction algorithms and the pseudorandomness notions
of leakage-resilience and graph regularity. We frame our investigation using
new, statistical distance-based variants of multicalibration that are closely
related to the concept of outcome indistinguishability. Adopting this
perspective leads us naturally not only to our graph theoretic results, but
also to new, more efficient algorithms for multicalibration in certain
parameter regimes and a novel proof of a hardcore lemma for real-valued
functions.
( 2
min )
In the paper, we propose a novel approach for solving Bayesian inverse
problems with physics-informed invertible neural networks (PI-INN). The
architecture of PI-INN consists of two sub-networks: an invertible neural
network (INN) and a neural basis network (NB-Net). The invertible map between
the parametric input and the INN output with the aid of NB-Net is constructed
to provide a tractable estimation of the posterior distribution, which enables
efficient sampling and accurate density evaluation. Furthermore, the loss
function of PI-INN includes two components: a residual-based physics-informed
loss term and a new independence loss term. The presented independence loss
term can Gaussianize the random latent variables and ensure statistical
independence between two parts of INN output by effectively utilizing the
estimated density function. Several numerical experiments are presented to
demonstrate the efficiency and accuracy of the proposed PI-INN, including
inverse kinematics, inverse problems of the 1-d and 2-d diffusion equations,
and seismic traveltime tomography.
( 2
min )
Previous studies have shown that leveraging domain index can significantly
boost domain adaptation performance (arXiv:2007.01807, arXiv:2202.03628).
However, such domain indices are not always available. To address this
challenge, we first provide a formal definition of domain index from the
probabilistic perspective, and then propose an adversarial variational Bayesian
framework that infers domain indices from multi-domain data, thereby providing
additional insight on domain relations and improving domain adaptation
performance. Our theoretical analysis shows that our adversarial variational
Bayesian framework finds the optimal domain index at equilibrium. Empirical
results on both synthetic and real data verify that our model can produce
interpretable domain indices which enable us to achieve superior performance
compared to state-of-the-art domain adaptation methods. Code is available at
https://github.com/Wang-ML-Lab/VDI.
( 2
min )
Modern machine learning systems are increasingly trained on large amounts of
data embedded in high-dimensional spaces. Often this is done without analyzing
the structure of the dataset. In this work, we propose a framework to study the
geometric structure of the data. We make use of our recently introduced
non-negative kernel (NNK) regression graphs to estimate the point density,
intrinsic dimension, and the linearity of the data manifold (curvature). We
further generalize the graph construction and geometric estimation to multiple
scale by iteratively merging neighborhoods in the input data. Our experiments
demonstrate the effectiveness of our proposed approach over other baselines in
estimating the local geometry of the data manifolds on synthetic and real
datasets.
( 2
min )
Motor brain-computer interface (BCI) development relies critically on neural
time series decoding algorithms. Recent advances in deep learning architectures
allow for automatic feature selection to approximate higher-order dependencies
in data. This article presents the FingerFlex model - a convolutional
encoder-decoder architecture adapted for finger movement regression on
electrocorticographic (ECoG) brain data. State-of-the-art performance was
achieved on a publicly available BCI competition IV dataset 4 with a
correlation coefficient between true and predicted trajectories up to 0.74. The
presented method provides the opportunity for developing fully-functional
high-precision cortical motor brain-computer interfaces.
( 2
min )
Hardware Trojans (HTs) are undesired design or manufacturing modifications
that can severely alter the security and functionality of digital integrated
circuits. HTs can be inserted according to various design criteria, e.g., nets
switching activity, observability, controllability, etc. However, to our
knowledge, most HT detection methods are only based on a single criterion,
i.e., nets switching activity. This paper proposes a multi-criteria
reinforcement learning (RL) HT detection tool that features a tunable reward
function for different HT detection scenarios. The tool allows for exploring
existing detection strategies and can adapt new detection scenarios with
minimal effort. We also propose a generic methodology for comparing HT
detection methods fairly. Our preliminary results show an average of 84.2%
successful HT detection in ISCAS-85 benchmark
( 2
min )
The proposed BSDE-based diffusion model represents a novel approach to
diffusion modeling, which extends the application of stochastic differential
equations (SDEs) in machine learning. Unlike traditional SDE-based diffusion
models, our model can determine the initial conditions necessary to reach a
desired terminal distribution by adapting an existing score function. We
demonstrate the theoretical guarantees of the model, the benefits of using
Lipschitz networks for score matching, and its potential applications in
various areas such as diffusion inversion, conditional diffusion, and
uncertainty quantification. Our work represents a contribution to the field of
score-based generative learning and offers a promising direction for solving
real-world problems.
( 2
min )
In this paper we present the Zeitview Rooftop Geometry (ZRG) dataset. ZRG
contains thousands of samples of high resolution orthomosaics of aerial imagery
of residential rooftops with corresponding digital surface models (DSM), 3D
rooftop wireframes, and multiview imagery generated point clouds for the
purpose of residential rooftop geometry and scene understanding. We perform
thorough benchmarks to illustrate the numerous applications unlocked by this
dataset and provide baselines for the tasks of roof outline extraction,
monocular height estimation, and planar roof structure extraction.
( 2
min )
We adapt reinforcement learning (RL) methods for continuous control to bridge
the gap between complete ignorance and perfect knowledge of the environment.
Our method, Partial Knowledge Least Squares Policy Iteration (PLSPI), takes
inspiration from both model-free RL and model-based control. It uses incomplete
information from a partial model and retains RL's data-driven adaption towards
optimal performance. The linear quadratic regulator provides a case study;
numerical experiments demonstrate the effectiveness and resulting benefits of
the proposed method.
( 2
min )
In this study, toward addressing the over-confident outputs of existing
artificial intelligence-based colorectal cancer (CRC) polyp classification
techniques, we propose a confidence-calibrated residual neural network.
Utilizing a novel vision-based tactile sensing (VS-TS) system and unique CRC
polyp phantoms, we demonstrate that traditional metrics such as accuracy and
precision are not sufficient to encapsulate model performance for handling a
sensitive CRC polyp diagnosis. To this end, we develop a residual neural
network classifier and address its over-confident outputs for CRC polyps
classification via the post-processing method of temperature scaling. To
evaluate the proposed method, we introduce noise and blur to the obtained
textural images of the VS-TS and test the model's reliability for non-ideal
inputs through reliability diagrams and other statistical metrics.
( 2
min )
Accurate detection of human presence in indoor environments is important for
various applications, such as energy management and security. In this paper, we
propose a novel system for human presence detection using the channel state
information (CSI) of WiFi signals. Our system named attention-enhanced deep
learning for presence detection (ALPD) employs an attention mechanism to
automatically select informative subcarriers from the CSI data and a
bidirectional long short-term memory (LSTM) network to capture temporal
dependencies in CSI. Additionally, we utilize a static feature to improve the
accuracy of human presence detection in static states. We evaluate the proposed
ALPD system by deploying a pair of WiFi access points (APs) for collecting CSI
dataset, which is further compared with several benchmarks. The results
demonstrate that our ALPD system outperforms the benchmarks in terms of
accuracy, especially in the presence of interference. Moreover, bidirectional
transmission data is beneficial to training improving stability and accuracy,
as well as reducing the costs of data collection for training. Overall, our
proposed ALPD system shows promising results for human presence detection using
WiFi CSI signals.
( 2
min )
The challenges faced by text classification with large tag systems in natural
language processing tasks include multiple tag systems, uneven data
distribution, and high noise. To address these problems, the ESimCSE
unsupervised comparative learning and UDA semi-supervised comparative learning
models are combined through the use of joint training techniques in the
models.The ESimCSE model efficiently learns text vector representations using
unlabeled data to achieve better classification results, while UDA is trained
using unlabeled data through semi-supervised learning methods to improve the
prediction performance of the models and stability, and further improve the
generalization ability of the model. In addition, adversarial training
techniques FGM and PGD are used in the model training process to improve the
robustness and reliability of the model. The experimental results show that
there is an 8% and 10% accuracy improvement relative to Baseline on the public
dataset Ruesters as well as on the operational dataset, respectively, and a 15%
improvement in manual validation accuracy can be achieved on the operational
dataset, indicating that the method is effective.
( 2
min )
We propose an experimental scheme for performing sensitive, high-precision
laser spectroscopy studies on fast exotic isotopes. By inducing a step-wise
resonant ionization of the atoms travelling inside an electric field and
subsequently detecting the ion and the corresponding electron, time- and
position-sensitive measurements of the resulting particles can be performed.
Using a Mixture Density Network (MDN), we can leverage this information to
predict the initial energy of individual atoms and thus apply a Doppler
correction of the observed transition frequencies on an event-by-event basis.
We conduct numerical simulations of the proposed experimental scheme and show
that kHz-level uncertainties can be achieved for ion beams produced at extreme
temperatures ($> 10^8$ K), with energy spreads as large as $10$ keV and
non-uniform velocity distributions. The ability to perform in-flight
spectroscopy, directly on highly energetic beams, offers unique opportunities
to studying short-lived isotopes with lifetimes in the millisecond range and
below, produced in low quantities, in hot and highly contaminated environments,
without the need for cooling techniques. Such species are of marked interest
for nuclear structure, astrophysics, and new physics searches.
( 2
min )
In this paper, we introduce a new nonlinear channel equalization method for
the coherent long-haul transmission based on Transformers. We show that due to
their capability to attend directly to the memory across a sequence of symbols,
Transformers can be used effectively with a parallelized structure. We present
an implementation of encoder part of Transformer for nonlinear equalization and
analyze its performance over a wide range of different hyper-parameters. It is
shown that by processing blocks of symbols at each iteration and carefully
selecting subsets of the encoder's output to be processed together, an
efficient nonlinear compensation can be achieved. We also propose the use of a
physic-informed mask inspired by nonlinear perturbation theory for reducing the
computational complexity of Transformer nonlinear equalization.
( 2
min )
In this paper, we investigate the robustness of an LSTM neural network
against noise injection attacks for electric load forecasting in an ideal
microgrid. The performance of the LSTM model is investigated under a black-box
Gaussian noise attack with different SNRs. It is assumed that attackers have
just access to the input data of the LSTM model. The results show that the
noise attack affects the performance of the LSTM model. The load prediction
means absolute error (MAE) is 0.047 MW for a healthy prediction, while this
value increases up to 0.097 MW for a Gaussian noise insertion with SNR= 6 dB.
To robustify the LSTM model against noise attack, a low-pass filter with
optimal cut-off frequency is applied at the model's input to remove the noise
attack. The filter performs better in case of noise with lower SNR and is less
promising for small noises.
( 2
min )
The average treatment effect, which is the difference in expectation of the
counterfactuals, is probably the most popular target effect in causal inference
with binary treatments. However, treatments may have effects beyond the mean,
for instance decreasing or increasing the variance. We propose a new
kernel-based test for distributional effects of the treatment. It is, to the
best of our knowledge, the first kernel-based, doubly-robust test with provably
valid type-I error. Furthermore, our proposed algorithm is efficient, avoiding
the use of permutations.
( 2
min )
Focusing on stochastic programming (SP) with covariate information, this
paper proposes an empirical risk minimization (ERM) method embedded within a
nonconvex piecewise affine decision rule (PADR), which aims to learn the direct
mapping from features to optimal decisions. We establish the nonasymptotic
consistency result of our PADR-based ERM model for unconstrained problems and
asymptotic consistency result for constrained ones. To solve the nonconvex and
nondifferentiable ERM problem, we develop an enhanced stochastic
majorization-minimization algorithm and establish the asymptotic convergence to
(composite strong) directional stationarity along with complexity analysis. We
show that the proposed PADR-based ERM method applies to a broad class of
nonconvex SP problems with theoretical consistency guarantees and computational
tractability. Our numerical study demonstrates the superior performance of
PADR-based ERM methods compared to state-of-the-art approaches under various
settings, with significantly lower costs, less computation time, and robustness
to feature dimensions and nonlinearity of the underlying dependency.
( 2
min )
We study Langevin-type algorithms for sampling from Gibbs distributions such
that the potentials are dissipative and their weak gradients have finite moduli
of continuity not necessarily convergent to zero. Our main result is a
non-asymptotic upper bound of the 2-Wasserstein distance between the Gibbs
distribution and the law of general Langevin-type algorithms based on the
Liptser--Shiryaev theory and Poincar\'{e} inequalities. We apply this bound to
show that the Langevin Monte Carlo algorithm can approximate Gibbs
distributions with arbitrary accuracy if the potentials are dissipative and
their gradients are uniformly continuous. We also propose Langevin-type
algorithms with spherical smoothing for potentials without convexity or
continuous differentiability.
( 2
min )
We describe a direct approach to estimate bipartite mutual information of a
classical spin system based on Monte Carlo sampling enhanced by autoregressive
neural networks. It allows studying arbitrary geometries of subsystems and can
be generalized to classical field theories. We demonstrate it on the Ising
model for four partitionings, including a multiply-connected even-odd division.
We show that the area law is satisfied for temperatures away from the critical
temperature: the constant term is universal, whereas the proportionality
coefficient is different for the even-odd partitioning.
( 2
min )
The Hierarchical Vote Collective of Transformation-based Ensembles
(HIVE-COTE) is a heterogeneous meta ensemble for time series classification.
Since it was first proposed in 2016, the algorithm has undergone some minor
changes and there is now a configurable, scalable and easy to use version
available in two open source repositories. We present an overview of the latest
stable HIVE-COTE, version 1.0, and describe how it differs to the original. We
provide a walkthrough guide of how to use the classifier, and conduct extensive
experimental evaluation of its predictive performance and resource usage. We
compare the performance of HIVE-COTE to three recently proposed algorithms
using the aeon toolkit.
( 2
min )
Forecast reconciliation is an important research topic. Yet, there is
currently neither formal framework nor practical method for the probabilistic
reconciliation of count time series. In this paper we propose a definition of
coherency and reconciled probabilistic forecast which applies to both
real-valued and count variables and a novel method for probabilistic
reconciliation. It is based on a generalization of Bayes' rule and it can
reconcile both real-value and count variables. When applied to count variables,
it yields a reconciled probability mass function. Our experiments with the
temporal reconciliation of count variables show a major forecast improvement
compared to the probabilistic Gaussian reconciliation.
( 2
min )
In this work, we study the performance of the Thompson Sampling algorithm for
Contextual Bandit problems based on the framework introduced by Neu et al. and
their concept of lifted information ratio. First, we prove a comprehensive
bound on the Thompson Sampling expected cumulative regret that depends on the
mutual information of the environment parameters and the history. Then, we
introduce new bounds on the lifted information ratio that hold for sub-Gaussian
rewards, thus generalizing the results from Neu et al. which analysis requires
binary rewards. Finally, we provide explicit regret bounds for the special
cases of unstructured bounded contextual bandits, structured bounded contextual
bandits with Laplace likelihood, structured Bernoulli bandits, and bounded
linear contextual bandits.
( 2
min )
In this article, we will research the Recommender System's implementation
about how it works and the algorithms used. We will explain the Recommender
System's algorithms based on mathematical principles, and find feasible methods
for improvements. The algorithms based on probability have its significance in
Recommender System, we will describe how they help to increase the accuracy and
speed of the algorithms. Both the weakness and the strength of two different
mathematical distance used to describe the similarity will be detailed
illustrated in this article.
( 2
min )
Do you need help to move your organization’s Machine Learning (ML) journey from pilot to production? You’re not alone. Most executives think ML can apply to any business decision, but on average only half of the ML projects make it to production. This post describes how to implement your first ML use case using Amazon […]
( 9
min )
Spotlighted by this week’s In the NVIDIA Studio featured artist Unmesh Dinda, NVIDIA Broadcast transforms the homes, apartments and dorm rooms of content creators, livestreamers and people working from home through the power of AI — all without the need for specialized equipment.
( 7
min )
Imagine a future where your vehicle’s interior offers personalized experiences and builds trust through human-machine interfaces (HMI) and AI. In this episode of the NVIDIA AI Podcast, Andreas Binner, chief technology officer at Rightware, delves into this fascinating topic with host Katie Burke Washabaugh. Rightware is a Helsinki-based company at the forefront of developing in-vehicle Read article >
( 5
min )
We recently introduced a new capability in the Amazon SageMaker Python SDK that lets data scientists run their machine learning (ML) code authored in their preferred integrated developer environment (IDE) and notebooks along with the associated runtime dependencies as Amazon SageMaker training jobs with minimal code changes to the experimentation done locally. Data scientists typically […]
( 13
min )
Many organizations use Gmail for their business email needs. Gmail for Business is part of Google Workspace, which provides a set of productivity and collaboration tools like Google Drive, Google Docs, Google Sheets, and more. For any organization, emails contain a wealth of information, which could be within the subject of an email, the message […]
( 9
min )
Announcements Tech Layoffs and Uncertainty Raise Big Questions for Higher Education Mass layoffs continue across the tech industry, with tens of thousands of workers losing their jobs in the first quarter of 2023. The reductions occurred from small startups to the biggest names in tech — Google, Amazon, Microsoft. Core technical roles such as data… Read More »DSC Weekly 25 April 2023 – Tech Layoffs and Uncertainty Raise Big Questions for Higher Education
The post DSC Weekly 25 April 2023 – Tech Layoffs and Uncertainty Raise Big Questions for Higher Education appeared first on Data Science Central.
( 19
min )
In recent years, businesses have been turning to artificial intelligence (AI) tools to improve their customer service and engagement. One such tool that has gained popularity is ChatGPT, an AI-powered chatbot that uses natural language processing (NLP) to engage with users and provide them with personalized responses. Integrating ChatGPT into your business’s app can bring… Read More »How Businesses can benefit by Integrating ChatGPT in their Apps
The post How Businesses can benefit by Integrating ChatGPT in their Apps appeared first on Data Science Central.
( 20
min )
Internal CPU Accelerators and HBM Enable Faster and Smarter HPC and AI Applications We have now entered the era when processor designers can leverage modular semiconductor manufacturing capabilities to speed frequently performed operations (such as small tensor operations) and offload a variety of housekeeping tasks (such as copying and zeroing memory) to dedicated on-chip accelerators. The… Read More »Internal CPU Accelerators and HBM Enable Faster and Smarter HPC and AI Applications
The post Internal CPU Accelerators and HBM Enable Faster and Smarter HPC and AI Applications appeared first on Data Science Central.
( 33
min )
Newly released open-source software can help developers guide generative AI applications to create impressive text responses that stay on track. NeMo Guardrails will help ensure smart applications powered by large language models (LLMs) are accurate, appropriate, on topic and secure. The software includes all the code, examples and documentation businesses need to add safety to Read article >
( 6
min )
ChatGPT users can now turn off chat history, allowing you to choose which conversations can be used to train our models.
( 2
min )
In the world of machine learning (ML), the quality of the dataset is of significant importance to model predictability. Although more data is usually better, large datasets with a high number of features can sometimes lead to non-optimal model performance due to the curse of dimensionality. Analysts can spend a significant amount of time transforming […]
( 9
min )
According to a PWC report, 32% of retail customers churn after one negative experience, and 73% of customers say that customer experience influences their purchase decisions. In the global retail industry, pre- and post-sales support are both important aspects of customer care. Numerous methods, including email, live chat, bots, and phone calls, are used to […]
( 8
min )
TLA+ is a high level, open-source, math-based language for modeling computer programs and systems–especially concurrent and distributed ones. It comes with tools to help eliminate fundamental design errors, which are hard to find and expensive to fix once they have been embedded in code or hardware. The TLA language was first published in 1993 by the […]
The post TLA+ Foundation aims to bring math-based software modeling to the mainstream appeared first on Microsoft Research.
( 9
min )
Along with Markov chain Monte Carlo (MCMC) methods, variational inference
(VI) has emerged as a central computational approach to large-scale Bayesian
inference. Rather than sampling from the true posterior $\pi$, VI aims at
producing a simple but effective approximation $\hat \pi$ to $\pi$ for which
summary statistics are easy to compute. However, unlike the well-studied MCMC
methodology, algorithmic guarantees for VI are still relatively less
well-understood. In this work, we propose principled methods for VI, in which
$\hat \pi$ is taken to be a Gaussian or a mixture of Gaussians, which rest upon
the theory of gradient flows on the Bures--Wasserstein space of Gaussian
measures. Akin to MCMC, it comes with strong theoretical guarantees when $\pi$
is log-concave.
( 2
min )
We consider using gradient descent to minimize the nonconvex function
$f(X)=\phi(XX^{T})$ over an $n\times r$ factor matrix $X$, in which $\phi$ is
an underlying smooth convex cost function defined over $n\times n$ matrices.
While only a second-order stationary point $X$ can be provably found in
reasonable time, if $X$ is additionally rank deficient, then its rank
deficiency certifies it as being globally optimal. This way of certifying
global optimality necessarily requires the search rank $r$ of the current
iterate $X$ to be overparameterized with respect to the rank $r^{\star}$ of the
global minimizer $X^{\star}$. Unfortunately, overparameterization significantly
slows down the convergence of gradient descent, from a linear rate with
$r=r^{\star}$ to a sublinear rate when $r>r^{\star}$, even when $\phi$ is
strongly convex. In this paper, we propose an inexpensive preconditioner that
restores the convergence rate of gradient descent back to linear in the
overparameterized case, while also making it agnostic to possible
ill-conditioning in the global minimizer $X^{\star}$.
( 2
min )
Neutron scattering experiments at three-axes spectrometers (TAS) investigate
magnetic and lattice excitations by measuring intensity distributions to
understand the origins of materials properties. The high demand and limited
availability of beam time for TAS experiments however raise the natural
question whether we can improve their efficiency and make better use of the
experimenter's time. In fact, there are a number of scientific problems that
require searching for signals, which may be time consuming and inefficient if
done manually due to measurements in uninformative regions. Here, we describe a
probabilistic active learning approach that not only runs autonomously, i.e.,
without human interference, but can also directly provide locations for
informative measurements in a mathematically sound and methodologically robust
way by exploiting log-Gaussian processes. Ultimately, the resulting benefits
can be demonstrated on a real TAS experiment and a benchmark including numerous
different excitations.
( 2
min )
Conservative inference is a major concern in simulation-based inference. It
has been shown that commonly used algorithms can produce overconfident
posterior approximations. Balancing has empirically proven to be an effective
way to mitigate this issue. However, its application remains limited to neural
ratio estimation. In this work, we extend balancing to any algorithm that
provides a posterior density. In particular, we introduce a balanced version of
both neural posterior estimation and contrastive neural ratio estimation. We
show empirically that the balanced versions tend to produce conservative
posterior approximations on a wide variety of benchmarks. In addition, we
provide an alternative interpretation of the balancing condition in terms of
the $\chi^2$ divergence.
( 2
min )
Recent breakthroughs in NLP largely increased the presence of ASR systems in
our daily lives. However, for many low-resource languages, ASR models still
need to be improved due in part to the difficulty of acquiring pertinent data.
This project aims to help advance research in ASR models for Swiss German
dialects, by providing insights about the performance of state-of-the-art ASR
models on recently published Swiss German speech datasets. We propose a novel
loss that takes into account the semantic distance between the predicted and
the ground-truth labels. We outperform current state-of-the-art results by
fine-tuning OpenAI's Whisper model on Swiss-German datasets.
( 2
min )
This article presents a leak localization methodology based on state
estimation and learning. The first is handled by an interpolation scheme,
whereas dictionary learning is considered for the second stage. The novel
proposed interpolation technique exploits the physics of the interconnections
between hydraulic heads of neighboring nodes in water distribution networks.
Additionally, residuals are directly interpolated instead of hydraulic head
values. The results of applying the proposed method to a well-known case study
(Modena) demonstrated the improvements of the new interpolation method with
respect to a state-of-the-art approach, both in terms of interpolation error
(considering state and residual estimation) and posterior localization.
( 2
min )
The use of machine learning (ML) inference for various applications is
growing drastically. ML inference services engage with users directly,
requiring fast and accurate responses. Moreover, these services face dynamic
workloads of requests, imposing changes in their computing resources. Failing
to right-size computing resources results in either latency service level
objectives (SLOs) violations or wasted computing resources. Adapting to dynamic
workloads considering all the pillars of accuracy, latency, and resource cost
is challenging. In response to these challenges, we propose InfAdapter, which
proactively selects a set of ML model variants with their resource allocations
to meet latency SLO while maximizing an objective function composed of accuracy
and cost. InfAdapter decreases SLO violation and costs up to 65% and 33%,
respectively, compared to a popular industry autoscaler (Kubernetes Vertical
Pod Autoscaler).
( 2
min )
Deploying machine learning models in production may allow adversaries to
infer sensitive information about training data. There is a vast literature
analyzing different types of inference risks, ranging from membership inference
to reconstruction attacks. Inspired by the success of games (i.e.,
probabilistic experiments) to study security properties in cryptography, some
authors describe privacy inference risks in machine learning using a similar
game-based style. However, adversary capabilities and goals are often stated in
subtly different ways from one presentation to the other, which makes it hard
to relate and compose results. In this paper, we present a game-based framework
to systematize the body of knowledge on privacy inference risks in machine
learning. We use this framework to (1) provide a unifying structure for
definitions of inference risks, (2) formally establish known relations among
definitions, and (3) to uncover hitherto unknown relations that would have been
difficult to spot otherwise.
( 2
min )
Hyperparameter optimization (HPO) is crucial for strong performance of deep
learning algorithms and real-world applications often impose some constraints,
such as memory usage, or latency on top of the performance requirement. In this
work, we propose constrained TPE (c-TPE), an extension of the widely-used
versatile Bayesian optimization method, tree-structured Parzen estimator (TPE),
to handle these constraints. Our proposed extension goes beyond a simple
combination of an existing acquisition function and the original TPE, and
instead includes modifications that address issues that cause poor performance.
We thoroughly analyze these modifications both empirically and theoretically,
providing insights into how they effectively overcome these challenges. In the
experiments, we demonstrate that c-TPE exhibits the best average rank
performance among existing methods with statistical significance on 81
expensive HPO settings.
( 2
min )
How do you scale a machine learning product at a startup? In particular, how
do you serve a greater volume, velocity, and variety of queries
cost-effectively? We break down costs into variable costs-the cost of serving
the model and performant-and fixed costs-the cost of developing and training
new models. We propose a framework for conceptualizing these costs, breaking
them into finer categories, and limn ways to reduce costs. Lastly, since in our
experience, the most expensive fixed cost of a machine learning system is the
cost of identifying the root causes of failures and driving continuous
improvement, we present a way to conceptualize the issues and share our
methodology for the same.
( 2
min )
We introduce a novel self-attention mechanism, which we call CSA (Chromatic
Self-Attention), which extends the notion of attention scores to attention
_filters_, independently modulating the feature channels. We showcase CSA in a
fully-attentional graph Transformer CGT (Chromatic Graph Transformer) which
integrates both graph structural information and edge features, completely
bypassing the need for local message-passing components. Our method flexibly
encodes graph structure through node-node interactions, by enriching the
original edge features with a relative positional encoding scheme. We propose a
new scheme based on random walks that encodes both structural and positional
information, and show how to incorporate higher-order topological information,
such as rings in molecular graphs. Our approach achieves state-of-the-art
results on the ZINC benchmark dataset, while providing a flexible framework for
encoding graph structure and incorporating higher-order topology.
( 2
min )
This article presents an identification benchmark based on data from a public
swimming pool in operation. Such a system is both a complex process and easily
understandable by all with regard to the stakes. Ultimately, the objective is
to reduce the energy bill while maintaining the level of quality of service.
This objective is general in scope and is not limited to public swimming pools.
This can be done effectively through what is known as economic predictive
control. This type of advanced control is based on a process model. It is the
aim of this article and the considered benchmark to show that such a dynamic
model can be obtained from operating data. For this, operational data is
formatted and shared, and model quality indicators are proposed. On this basis,
the first identification results illustrate the results obtained by a linear
multivariable model on the one hand, and by a neural dynamic model on the other
hand. The benchmark calls for other proposals and results from control and data
scientists for comparison.
( 2
min )
This short note describes and proves a connectedness property which was
introduced in Blocher et al. [2023] in the context of data depth functions for
partial orders. The connectedness property gives a structural insight into
union-free generic sets. These sets, presented in Blocher et al. [2023], are
defined by using a closure operator on the set of all partial orders which
naturally appears within the theory of formal concept analysis. In the language
of formal concept analysis, the property of connectedness can be vividly
proven. However, since within Blocher et al. [2023] we did not discuss formal
concept analysis, we outsourced the proof to this note.
( 2
min )
Variational autoencoders (VAEs) are used for transfer learning across various
research domains such as music generation or medical image analysis. However,
there is no principled way to assess before transfer which components to
retrain or whether transfer learning is likely to help on a target task. We
propose to explore this question through the lens of representational
similarity. Specifically, using Centred Kernel Alignment (CKA) to evaluate the
similarity of VAEs trained on different datasets, we show that encoders'
representations are generic but decoders' specific. Based on these insights, we
discuss the implications for selecting which components of a VAE to retrain and
propose a method to visually assess whether transfer learning is likely to help
on classification tasks.
( 2
min )
Exploration is a fundamental aspect of reinforcement learning (RL), and its
effectiveness crucially decides the performance of RL algorithms, especially
when facing sparse extrinsic rewards. Recent studies showed the effectiveness
of encouraging exploration with intrinsic rewards estimated from novelty in
observations. However, there is a gap between the novelty of an observation and
an exploration in general, because the stochasticity in the environment as well
as the behavior of an agent may affect the observation. To estimate exploratory
behaviors accurately, we propose DEIR, a novel method where we theoretically
derive an intrinsic reward from a conditional mutual information term that
principally scales with the novelty contributed by agent explorations, and
materialize the reward with a discriminative forward model. We conduct
extensive experiments in both standard and hardened exploration games in
MiniGrid to show that DEIR quickly learns a better policy than baselines. Our
evaluations in ProcGen demonstrate both generalization capabilities and the
general applicability of our intrinsic reward.
( 2
min )
Recent years have seen a rich literature of data-driven approaches designed
for power grid applications. However, insufficient consideration of domain
knowledge can impose a high risk to the practicality of the methods.
Specifically, ignoring the grid-specific spatiotemporal patterns (in load,
generation, and topology, etc.) can lead to outputting infeasible,
unrealizable, or completely meaningless predictions on new inputs. To address
this concern, this paper investigates real-world operational data to provide
insights into power grid behavioral patterns, including the time-varying
topology, load, and generation, as well as the spatial differences (in peak
hours, diverse styles) between individual loads and generations. Then based on
these observations, we evaluate the generalization risks in some existing ML
works causedby ignoring these grid-specific patterns in model design and
training.
( 2
min )
It is difficult to identify anomalies in time series, especially when there
is a lot of noise. Denoising techniques can remove the noise but this technique
can cause a significant loss of information. To detect anomalies in the time
series we have proposed an attention free conditional autoencoder (AF-CA). We
started from the autoencoder conditional model on which we added an
Attention-Free LSTM layer \cite{inzirillo2022attention} in order to make the
anomaly detection capacity more reliable and to increase the power of anomaly
detection. We compared the results of our Attention Free Conditional
Autoencoder with those of an LSTM Autoencoder and clearly improved the
explanatory power of the model and therefore the detection of anomaly in noisy
time series.
( 2
min )
This article measures how sparsity can make neural networks more robust to
membership inference attacks. The obtained empirical results show that sparsity
improves the privacy of the network, while preserving comparable performances
on the task at hand. This empirical study completes and extends existing
literature.
( 2
min )
In Part I of the series “Creating Healthy AI Utility Function: Importance of Diversity,” I talked about the importance of embracing conflict and diversity to create a Healthy AI Utility Function; that is, creating an AI Utility Function that continuously balances conflicting KPIs and metrics to deliver responsible and ethical outcomes. The AI Utility Function… Read More »Creating Healthy AI Utility Function: ChatGPT Example – Part II
The post Creating Healthy AI Utility Function: ChatGPT Example – Part II appeared first on Data Science Central.
( 21
min )
submitted by /u/Joffylad
[link] [comments]
( 43
min )
Just in last 1 year, top 0.1% saw their wealth increase by 6 trillion dollars, bigger than wealth of most countries. https://www.cnbc.com/amp/2022/04/01/richest-one-percent-gained-trillions-in-wealth-2021.html
submitted by /u/timesarewasting
[link] [comments]
( 43
min )
For those of you interested in diving into the future of AI with some of the worlds leading AI experts, my company is hosting this free virtual event.
Kris Hammond (advises the U.N. and White House on AI) and his Northwestern students built us a custom AI/deepfake chat bot that will actually be on the panel answering questions and engaging in discussion…talk about Black Mirror situations. It should get interesting.
For those getting into AI or that understand how important it is for remaining competitive in your career, you should def check it out.
Here’s a link: https://chicagoinnovation.com/events/ai-vs-iq/
submitted by /u/chickenfettuccine
[link] [comments]
( 43
min )
submitted by /u/cgwuaqueduct
[link] [comments]
( 43
min )
submitted by /u/saintshing
[link] [comments]
( 43
min )
submitted by /u/nickb
[link] [comments]
( 41
min )
With the advances of IoT developments, copious sensor data are communicated
through wireless networks and create the opportunity of building Digital Twins
to mirror and simulate the complex physical world. Digital Twin has long been
believed to rely heavily on domain knowledge, but we argue that this leads to a
high barrier of entry and slow development due to the scarcity and cost of
human experts. In this paper, we propose Digital Twin Graph (DTG), a general
data structure associated with a processing framework that constructs digital
twins in a fully automated and domain-agnostic manner. This work represents the
first effort that takes a completely data-driven and (unconventional) graph
learning approach to addresses key digital twin challenges.
( 2
min )
This study proposes a deep learning model for the classification and
segmentation of brain tumors from magnetic resonance imaging (MRI) scans. The
classification model is based on the EfficientNetB1 architecture and is trained
to classify images into four classes: meningioma, glioma, pituitary adenoma,
and no tumor. The segmentation model is based on the U-Net architecture and is
trained to accurately segment the tumor from the MRI images. The models are
evaluated on a publicly available dataset and achieve high accuracy and
segmentation metrics, indicating their potential for clinical use in the
diagnosis and treatment of brain tumors.
( 2
min )
Questions remain on the robustness of data-driven learning methods when
crossing the gap from simulation to reality. We utilize weight anchoring, a
method known from continual learning, to cultivate and fixate desired behavior
in Neural Networks. Weight anchoring may be used to find a solution to a
learning problem that is nearby the solution of another learning problem.
Thereby, learning can be carried out in optimal environments without neglecting
or unlearning desired behavior. We demonstrate this approach on the example of
learning mixed QoS-efficient discrete resource scheduling with infrequent
priority messages. Results show that this method provides performance
comparable to the state of the art of augmenting a simulation environment,
alongside significantly increased robustness and steerability.
( 2
min )
This work brings the leading accuracy, sample efficiency, and robustness of
deep equivariant neural networks to the extreme computational scale. This is
achieved through a combination of innovative model architecture, massive
parallelization, and models and implementations optimized for efficient GPU
utilization. The resulting Allegro architecture bridges the accuracy-speed
tradeoff of atomistic simulations and enables description of dynamics in
structures of unprecedented complexity at quantum fidelity. To illustrate the
scalability of Allegro, we perform nanoseconds-long stable simulations of
protein dynamics and scale up to a 44-million atom structure of a complete,
all-atom, explicitly solvated HIV capsid on the Perlmutter supercomputer. We
demonstrate excellent strong scaling up to 100 million atoms and 70% weak
scaling to 5120 A100 GPUs.
( 2
min )
The K Nearest Neighbors (KNN) classifier is widely used in many fields such
as fingerprint-based localization or medicine. It determines the class
membership of unlabelled sample based on the class memberships of the K
labelled samples, the so-called nearest neighbors, that are closest to the
unlabelled sample. The choice of K has been the topic of various studies and
proposed KNN-variants. Yet no variant has been proven to outperform all other
variants. In this paper a new KNN-variant is proposed which ensures that the K
nearest neighbors are indeed close to the unlabelled sample and finds K along
the way. The proposed algorithm is tested and compared to the standard KNN in
theoretical scenarios and for indoor localization based on ion-mobility
spectrometry fingerprints. It achieves a higher classification accuracy than
the KNN in the tests, while requiring having the same computational demand.
( 2
min )
Kernel-based modal statistical methods include mode estimation, regression,
and clustering. Estimation accuracy of these methods depends on the kernel used
as well as the bandwidth. We study effect of the selection of the kernel
function to the estimation accuracy of these methods. In particular, we
theoretically show a (multivariate) optimal kernel that minimizes its
analytically-obtained asymptotic error criterion when using an optimal
bandwidth, among a certain kernel class defined via the number of its sign
changes.
( 2
min )
Quantum computation has a strong implication for advancing the current
limitation of machine learning algorithms to deal with higher data dimensions
or reducing the overall training parameters for a deep neural network model.
Based on a gate-based quantum computer, a parameterized quantum circuit was
designed to solve a model-free reinforcement learning problem with the deep-Q
learning method. This research has investigated and evaluated its potential.
Therefore, a novel PQC based on the latest Qiskit and PyTorch framework was
designed and trained to compare with a full-classical deep neural network with
and without integrated PQC. At the end of the research, the research draws its
conclusion and prospects on developing deep quantum learning in solving a maze
problem or other reinforcement learning problems.
( 2
min )
This paper presents two novel deterministic initialization procedures for
K-means clustering based on a modified crowding distance. The procedures, named
CKmeans and FCKmeans, use more crowded points as initial centroids.
Experimental studies on multiple datasets demonstrate that the proposed
approach outperforms Kmeans and Kmeans++ in terms of clustering accuracy. The
effectiveness of CKmeans and FCKmeans is attributed to their ability to select
better initial centroids based on the modified crowding distance. Overall, the
proposed approach provides a promising alternative for improving K-means
clustering.
( 2
min )
We used survival analysis to quantify the impact of postdischarge evaluation
and management (E/M) services in preventing hospital readmission or death. Our
approach avoids a specific pitfall of applying machine learning to this
problem, which is an inflated estimate of the effect of interventions, due to
survivors bias -- where the magnitude of inflation may be conditional on
heterogeneous confounders in the population. This bias arises simply because in
order to receive an intervention after discharge, a person must not have been
readmitted in the intervening period. After deriving an expression for this
phantom effect, we controlled for this and other biases within an inherently
interpretable Bayesian survival framework. We identified case management
services as being the most impactful for reducing readmissions overall,
particularly for patients discharged to long term care facilities, with high
resource utilization in the quarter preceding admission.
( 2
min )
We study the impacts of business cycles on machine learning (ML) predictions.
Using the S&P 500 index, we find that ML models perform worse during most
recessions, and the inclusion of recession history or the risk-free rate does
not necessarily improve their performance. Investigating recessions where
models perform well, we find that they exhibit lower market volatility than
other recessions. This implies that the improved performance is not due to the
merit of ML methods but rather factors such as effective monetary policies that
stabilized the market. We recommend that ML practitioners evaluate their models
during both recessions and expansions.
( 2
min )
We propose a framework for descriptively analyzing sets of partial orders
based on the concept of depth functions. Despite intensive studies of depth
functions in linear and metric spaces, there is very little discussion on depth
functions for non-standard data types such as partial orders. We introduce an
adaptation of the well-known simplicial depth to the set of all partial orders,
the union-free generic (ufg) depth. Moreover, we utilize our ufg depth for a
comparison of machine learning algorithms based on multidimensional performance
measures. Concretely, we analyze the distribution of different classifier
performances over a sample of standard benchmark data sets. Our results
promisingly demonstrate that our approach differs substantially from existing
benchmarking approaches and, therefore, adds a new perspective to the vivid
debate on the comparison of classifiers.
( 2
min )
We present a theory for the previously unexplained divergent behavior noticed
in the training of large language models. We argue that the phenomenon is an
artifact of the dominant optimization algorithm used for training, called Adam.
We observe that Adam can enter a state in which the parameter update vector has
a relatively large norm and is essentially uncorrelated with the direction of
descent on the training loss landscape, leading to divergence. This artifact is
more likely to be observed in the training of a deep model with a large batch
size, which is the typical setting of large-scale language model training. To
argue the theory, we present observations from the training runs of the
language models of different scales: 7 billion, 30 billion, 65 billion, and 546
billion parameters.
( 2
min )
Support vector clustering is an important clustering method. However, it
suffers from a scalability issue due to its computational expensive cluster
assignment step. In this paper we accelertate the support vector clustering via
spectrum-preserving data compression. Specifically, we first compress the
original data set into a small amount of spectrally representative aggregated
data points. Then, we perform standard support vector clustering on the
compressed data set. Finally, we map the clustering results of the compressed
data set back to discover the clusters in the original data set. Our extensive
experimental results on real-world data set demonstrate dramatically speedups
over standard support vector clustering without sacrificing clustering quality.
( 2
min )
Support vector clustering is an important clustering method. However, it
suffers from a scalability issue due to its computational expensive cluster
assignment step. In this paper we accelertate the support vector clustering via
spectrum-preserving data compression. Specifically, we first compress the
original data set into a small amount of spectrally representative aggregated
data points. Then, we perform standard support vector clustering on the
compressed data set. Finally, we map the clustering results of the compressed
data set back to discover the clusters in the original data set. Our extensive
experimental results on real-world data set demonstrate dramatically speedups
over standard support vector clustering without sacrificing clustering quality.
( 2
min )
There isn’t a foolproof formula for building a successful digital firm — the risk of starting a business is high. There’s more to the frequently cited statistic that nine out of ten companies fail — a reason you should check out this step-by-step guide to starting a successful startup. The COVID-19 pandemic has put pressure… Read More »5 Crucial Steps To Starting A Successful Hi-Tech Startup: From Idea To Promotion
The post 5 Crucial Steps To Starting A Successful Hi-Tech Startup: From Idea To Promotion appeared first on Data Science Central.
( 21
min )
From climate modeling to endangered species conservation, developers, researchers and companies are keeping an AI on the environment with the help of NVIDIA technology. They’re using NVIDIA GPUs and software to track endangered African black rhinos, forecast the availability of solar energy in the U.K., build detailed climate models and monitor environmental disasters from satellite Read article >
( 7
min )
Content creators using Epic Games’ open, advanced real-time 3D creation tool, Unreal Engine, are now equipped with more features to bring their work to life with NVIDIA Omniverse, a platform for creating and operating metaverse applications. The Omniverse Connector for Unreal Engine’s 201.0 update brings significant enhancements to creative workflows using both open platforms. Streamlining Read article >
( 6
min )
What’s the difference between NVIDIA GeForce RTX 30 and 40 Series GPUs for gamers? To briefly set aside the technical specifications, the difference lies in the level of performance and capability each series offers. Both deliver great graphics. Both offer advanced new features driven by NVIDIA’s global AI revolution a decade ago. Either can power Read article >
( 6
min )
Batch inference is a common pattern where prediction requests are batched together on input, a job runs to process those requests against a trained model, and the output includes batch prediction responses that can then be consumed by other applications or business functions. Running batch use cases in production environments requires a repeatable process for […]
( 14
min )
The technology of MIT alumni-founded Hosta a.i. creates detailed property assessments from photos.
( 9
min )
submitted by /u/Ad3t0
[link] [comments]
( 42
min )
submitted by /u/DarkangelUK
[link] [comments]
( 43
min )
submitted by /u/Sparkvoltage
[link] [comments]
( 43
min )
Hello! Not sure if this is the right place to ask.
I am working on a startup, I was wondering what people think are some gaps in current machine learning infrastructure solutions like WandB, or Neptune.ai.
I'd love to know what people think are some missing features for products like these, or what completely new features they would like to see!
submitted by /u/spirited__tree
[link] [comments]
( 43
min )
Hi all,
Hope you are all well. Last time I posted about the fastLLaMa project on here, I had a lot of support from you guys and I really appreciated it. Motivated me to try random experiments and new things!
Thought I would give an update after a month.
Yesterday we added support to enable users to attach and detach LoRA adapters quickly during the runtime. This work was built on top of the original llama.cpp repo with some modifications that impact the adapter size (We are figuring out ways to reduce the adapter size through possible quantization).
We also built on top of our save load feature to enable quick context switching during run time! This should enable a single running instance to server multiple sessions.
We were also grateful for the feature requests from the last post a…
( 46
min )
More than 50 automotive companies around the world have deployed over 800 autonomous test vehicles powered by the NVIDIA DRIVE Hyperion automotive compute architecture, which has recently achieved new safety milestones. The latest NVIDIA DRIVE Hyperion architecture is based on the DRIVE Orin system-on-a-chip (SoC). Many NVIDIA DRIVE processes, as well as hardware and software Read article >
( 5
min )
GFN Thursday rolls up this week with a hot new deal for a GeForce NOW six-month Priority membership. Enjoy the cloud gaming service with seven new games to stream this week, including more favorites from Bandai Namco Europe and F1 2021 from Electronic Arts. Make Gaming a Priority Starting today, GeForce NOW is offering a Read article >
( 6
min )
NVIDIA today recognized a dozen partners for their work helping customers in Europe, the Middle East and Africa harness the power of AI across industries. At a virtual EMEA Partner Day event, which was hosted by the NVIDIA Partner Network (NPN) and drew more than 750 registrants, Partner of the Year awards were given to Read article >
( 6
min )
Each machine learning (ML) system has a unique service level agreement (SLA) requirement with respect to latency, throughput, and cost metrics. With advancements in hardware design, a wide range of CPU- and GPU-based infrastructures are available to help you speed up inference performance. Also, you can build these ML systems with a combination of ML […]
( 11
min )
These tunable proteins could be used to create new materials with specific mechanical properties, like toughness or flexibility.
( 10
min )
This study introduces and investigates the capabilities of three different
text mining approaches, namely Latent Semantic Analysis, Latent Dirichlet
Analysis, and Clustering Word Vectors, for automating code extraction from a
relatively small discussion board dataset. We compare the outputs of each
algorithm with a previous dataset that was manually coded by two human raters.
The results show that even with a relatively small dataset, automated
approaches can be an asset to course instructors by extracting some of the
discussion codes, which can be used in Epistemic Network Analysis.
( 2
min )
Mining data streams is one of the main studies in machine learning area due
to its application in many knowledge areas. One of the major challenges on
mining data streams is concept drift, which requires the learner to discard the
current concept and adapt to a new one. Ensemble-based drift detection
algorithms have been used successfully to the classification task but usually
maintain a fixed size ensemble of learners running the risk of needlessly
spending processing time and memory. In this paper we present improvements to
the Scale-free Network Regressor (SFNR), a dynamic ensemble-based method for
regression that employs social networks theory. In order to detect concept
drifts SFNR uses the Adaptive Window (ADWIN) algorithm. Results show
improvements in accuracy, especially in concept drift situations and better
performance compared to other state-of-the-art algorithms in both real and
synthetic data.
( 2
min )
The quality of air is closely linked with the life quality of humans,
plantations, and wildlife. It needs to be monitored and preserved continuously.
Transportations, industries, construction sites, generators, fireworks, and
waste burning have a major percentage in degrading the air quality. These
sources are required to be used in a safe and controlled manner. Using
traditional laboratory analysis or installing bulk and expensive models every
few miles is no longer efficient. Smart devices are needed for collecting and
analyzing air data. The quality of air depends on various factors, including
location, traffic, and time. Recent researches are using machine learning
algorithms, big data technologies, and the Internet of Things to propose a
stable and efficient model for the stated purpose. This review paper focuses on
studying and compiling recent research in this field and emphasizes the Data
sources, Monitoring, and Forecasting models. The main objective of this paper
is to provide the astuteness of the researches happening to improve the various
aspects of air polluting models. Further, it casts light on the various
research issues and challenges also.
( 2
min )
Successful deployment of artificial intelligence (AI) in various settings has
led to numerous positive outcomes for individuals and society. However, AI
systems have also been shown to harm parts of the population due to biased
predictions. We take a closer look at AI fairness and analyse how lack of AI
fairness can lead to deepening of biases over time and act as a social
stressor. If the issues persist, it could have undesirable long-term
implications on society, reinforced by interactions with other risks. We
examine current strategies for improving AI fairness, assess their limitations
in terms of real-world deployment, and explore potential paths forward to
ensure we reap AI's benefits without harming significant parts of the society.
( 2
min )
Advances in mobile communication capabilities open the door for closer
integration of pre-hospital and in-hospital care processes. For example,
medical specialists can be enabled to guide on-site paramedics and can, in
turn, be supplied with live vitals or visuals. Consolidating such
performance-critical applications with the highly complex workings of mobile
communications requires solutions both reliable and efficient, yet easy to
integrate with existing systems. This paper explores the application of Deep
Deterministic Policy Gradient~(\ddpg) methods for learning a communications
resource scheduling algorithm with special regards to priority users. Unlike
the popular Deep-Q-Network methods, the \ddpg is able to produce
continuous-valued output. With light post-processing, the resulting scheduler
is able to achieve high performance on a flexible sum-utility goal.
( 2
min )
We study the training dynamics of shallow neural networks, in a two-timescale
regime in which the stepsizes for the inner layer are much smaller than those
for the outer layer. In this regime, we prove convergence of the gradient flow
to a global optimum of the non-convex optimization problem in a simple
univariate setting. The number of neurons need not be asymptotically large for
our result to hold, distinguishing our result from popular recent approaches
such as the neural tangent kernel or mean-field regimes. Experimental
illustration is provided, showing that the stochastic gradient descent behaves
according to our description of the gradient flow and thus converges to a
global optimum in the two-timescale regime, but can fail outside of this
regime.
( 2
min )
This paper introduces the QDQN-DPER framework to enhance the efficiency of
quantum reinforcement learning (QRL) in solving sequential decision tasks. The
framework incorporates prioritized experience replay and asynchronous training
into the training algorithm to reduce the high sampling complexities. Numerical
simulations demonstrate that QDQN-DPER outperforms the baseline distributed
quantum Q learning with the same model architecture. The proposed framework
holds potential for more complex tasks while maintaining training efficiency.
( 2
min )
We discuss the discontinuities that arise when mapping unordered objects to
neural network outputs of fixed permutation, referred to as the responsibility
problem. Prior work has proved the existence of the issue by identifying a
single discontinuity. Here, we show that discontinuities under such models are
uncountably infinite, motivating further research into neural networks for
unordered data.
( 2
min )
Prompt-based learning reformulates downstream tasks as cloze problems by
combining the original input with a template. This technique is particularly
useful in few-shot learning, where a model is trained on a limited amount of
data. However, the limited templates and text used in few-shot prompt-based
learning still leave significant room for performance improvement.
Additionally, existing methods using model ensembles can constrain the model
efficiency. To address these issues, we propose an augmentation method called
MixPro, which augments both the vanilla input text and the templates through
token-level, sentence-level, and epoch-level Mixup strategies. We conduct
experiments on five few-shot datasets, and the results show that MixPro
outperforms other augmentation baselines, improving model performance by an
average of 5.08% compared to before augmentation.
( 2
min )
Multicalibration is a notion of fairness that aims to provide accurate
predictions across a large set of groups. Multicalibration is known to be a
different goal than loss minimization, even for simple predictors such as
linear functions. In this note, we show that for (almost all) large neural
network sizes, optimally minimizing squared error leads to multicalibration.
Our results are about representational aspects of neural networks, and not
about algorithmic or sample complexity considerations. Previous such results
were known only for predictors that were nearly Bayes-optimal and were
therefore representation independent. We emphasize that our results do not
apply to specific algorithms for optimizing neural networks, such as SGD, and
they should not be interpreted as "fairness comes for free from optimizing
neural networks".
( 2
min )
Many machine learning methods assume that the training and test data follow
the same distribution. However, in the real world, this assumption is very
often violated. In particular, the phenomenon that the marginal distribution of
the data changes is called covariate shift, one of the most important research
topics in machine learning. We show that the well-known family of covariate
shift adaptation methods is unified in the framework of information geometry.
Furthermore, we show that parameter search for geometrically generalized
covariate shift adaptation method can be achieved efficiently. Numerical
experiments show that our generalization can achieve better performance than
the existing methods it encompasses.
( 2
min )
The recent advances in representation learning inspire us to take on the
challenging problem of unsupervised image classification tasks in a principled
way. We propose ContraCluster, an unsupervised image classification method that
combines clustering with the power of contrastive self-supervised learning.
ContraCluster consists of three stages: (1) contrastive self-supervised
pre-training (CPT), (2) contrastive prototype sampling (CPS), and (3)
prototype-based semi-supervised fine-tuning (PB-SFT). CPS can select highly
accurate, categorically prototypical images in an embedding space learned by
contrastive learning. We use sampled prototypes as noisy labeled data to
perform semi-supervised fine-tuning (PB-SFT), leveraging small prototypes and
large unlabeled data to further enhance the accuracy. We demonstrate
empirically that ContraCluster achieves new state-of-the-art results for
standard benchmark datasets including CIFAR-10, STL-10, and ImageNet-10. For
example, ContraCluster achieves about 90.8% accuracy for CIFAR-10, which
outperforms DAC (52.2%), IIC (61.7%), and SCAN (87.6%) by a large margin.
Without any labels, ContraCluster can achieve a 90.8% accuracy that is
comparable to 95.8% by the best supervised counterpart.
( 2
min )
Sea surface temperature (SST) is uniquely important to the Earth's atmosphere
since its dynamics are a major force in shaping local and global climate and
profoundly affect our ecosystems. Accurate forecasting of SST brings
significant economic and social implications, for example, better preparation
for extreme weather such as severe droughts or tropical cyclones months ahead.
However, such a task faces unique challenges due to the intrinsic complexity
and uncertainty of ocean systems. Recently, deep learning techniques, such as
graphical neural networks (GNN), have been applied to address this task. Even
though these methods have some success, they frequently have serious drawbacks
when it comes to investigating dynamic spatiotemporal dependencies between
signals. To solve this problem, this paper proposes a novel static and dynamic
learnable personalized graph convolution network (SD-LPGC). Specifically, two
graph learning layers are first constructed to respectively model the stable
long-term and short-term evolutionary patterns hidden in the multivariate SST
signals. Then, a learnable personalized convolution layer is designed to fuse
this information. Our experiments on real SST datasets demonstrate the
state-of-the-art performances of the proposed approach on the forecasting task.
( 2
min )
Federated Learning (FL) aims to train a machine learning (ML) model in a
distributed fashion to strengthen data privacy with limited data migration
costs. It is a distributed learning framework naturally suitable for
privacy-sensitive medical imaging datasets. However, most current FL-based
medical imaging works assume silos have ground truth labels for training. In
practice, label acquisition in the medical field is challenging as it often
requires extensive labor and time costs. To address this challenge and leverage
the unannotated data silos to improve modeling, we propose an alternate
training-based framework, Federated Alternate Training (FAT), that alters
training between annotated data silos and unannotated data silos. Annotated
data silos exploit annotations to learn a reasonable global segmentation model.
Meanwhile, unannotated data silos use the global segmentation model as a target
model to generate pseudo labels for self-supervised learning. We evaluate the
performance of the proposed framework on two naturally partitioned Federated
datasets, KiTS19 and FeTS2021, and show its promising performance.
( 2
min )
Parkinson's disease (PD) has been found to affect 1 out of every 1000 people,
being more inclined towards the population above 60 years. Leveraging
wearable-systems to find accurate biomarkers for diagnosis has become the need
of the hour, especially for a neurodegenerative condition like Parkinson's.
This work aims at focusing on early-occurring, common symptoms, such as motor
and gait related parameters to arrive at a quantitative analysis on the
feasibility of an economical and a robust wearable device. A subset of the
Parkinson's Progression Markers Initiative (PPMI), PPMI Gait dataset has been
utilised for feature-selection after a thorough analysis with various Machine
Learning algorithms. Identified influential features has then been used to test
real-time data for early detection of Parkinson Syndrome, with a model accuracy
of 91.9%
( 2
min )
We apply Bayesian optimization and reinforcement learning to a problem in
topology: the question of when a knot bounds a ribbon disk. This question is
relevant in an approach to disproving the four-dimensional smooth Poincar\'e
conjecture; using our programs, we rule out many potential counterexamples to
the conjecture. We also show that the programs are successful in detecting many
ribbon knots in the range of up to 70 crossings.
( 2
min )
Precise estimation of cross-correlation or similarity between two random
variables lies at the heart of signal detection, hyperdimensional computing,
associative memories, and neural networks. Although a vast literature exists on
different methods for estimating cross-correlations, the question what is the
best and simplest method to estimate cross-correlations using finite samples ?
is still not clear. In this paper, we first argue that the standard empirical
approach might not be the optimal method even though the estimator exhibits
uniform convergence to the true cross-correlation. Instead, we show that there
exists a large class of simple non-linear functions that can be used to
construct cross-correlators with a higher signal-to-noise ratio (SNR). To
demonstrate this, we first present a general mathematical framework using
Price's Theorem that allows us to analyze cross-correlators constructed using a
mixture of piece-wise linear functions. Using this framework and
high-dimensional embedding, we show that some of the most promising
cross-correlators are based on Huber's loss functions, margin-propagation (MP)
functions, and the log-sum-exp functions.
( 2
min )
We extend the global convergence result of Chatterjee
\cite{chatterjee2022convergence} by considering the stochastic gradient descent
(SGD) for non-convex objective functions. With minimal additional assumptions
that can be realized by finitely wide neural networks, we prove that if we
initialize inside a local region where the \L{}ajasiewicz condition holds, with
a positive probability, the stochastic gradient iterates converge to a global
minimum inside this region. A key component of our proof is to ensure that the
whole trajectories of SGD stay inside the local region with a positive
probability. For that, we assume the SGD noise scales with the objective
function, which is called machine learning noise and achievable in many real
examples. Furthermore, we provide a negative argument to show why using the
boundedness of noise with Robbins-Monro type step sizes is not enough to keep
the key component valid.
( 2
min )
Spatiotemporal (ST) data collected by sensors can be represented as
multi-variate time series, which is a sequence of data points listed in an
order of time. Despite the vast amount of useful information, the ST data
usually suffer from the issue of missing or incomplete data, which also limits
its applications. Imputation is one viable solution and is often used to
prepossess the data for further applications. However, in practice, n practice,
spatiotemporal data imputation is quite difficult due to the complexity of
spatiotemporal dependencies with dynamic changes in the traffic network and is
a crucial prepossessing task for further applications. Existing approaches
mostly only capture the temporal dependencies in time series or static spatial
dependencies. They fail to directly model the spatiotemporal dependencies, and
the representation ability of the models is relatively limited.
( 2
min )
Running complex sets of machine learning experiments is challenging and
time-consuming due to the lack of a unified framework. This leaves researchers
forced to spend time implementing necessary features such as parallelization,
caching, and checkpointing themselves instead of focussing on their project. To
simplify the process, in this paper, we introduce Memento, a Python package
that is designed to aid researchers and data scientists in the efficient
management and execution of computationally intensive experiments. Memento has
the capacity to streamline any experimental pipeline by providing a
straightforward configuration matrix and the ability to concurrently run
experiments across multiple threads. A demonstration of Memento is available
at: https://wickerlab.org/publication/memento.
( 2
min )
Multicalibration is a notion of fairness that aims to provide accurate
predictions across a large set of groups. Multicalibration is known to be a
different goal than loss minimization, even for simple predictors such as
linear functions. In this note, we show that for (almost all) large neural
network sizes, optimally minimizing squared error leads to multicalibration.
Our results are about representational aspects of neural networks, and not
about algorithmic or sample complexity considerations. Previous such results
were known only for predictors that were nearly Bayes-optimal and were
therefore representation independent. We emphasize that our results do not
apply to specific algorithms for optimizing neural networks, such as SGD, and
they should not be interpreted as "fairness comes for free from optimizing
neural networks".
( 2
min )
Outliers widely occur in big-data applications and may severely affect
statistical estimation and inference. In this paper, a framework of
outlier-resistant estimation is introduced to robustify an arbitrarily given
loss function. It has a close connection to the method of trimming and includes
explicit outlyingness parameters for all samples, which in turn facilitates
computation, theory, and parameter tuning. To tackle the issues of nonconvexity
and nonsmoothness, we develop scalable algorithms with implementation ease and
guaranteed fast convergence. In particular, a new technique is proposed to
alleviate the requirement on the starting point such that on regular datasets,
the number of data resamplings can be substantially reduced. Based on combined
statistical and computational treatments, we are able to perform nonasymptotic
analysis beyond M-estimation. The obtained resistant estimators, though not
necessarily globally or even locally optimal, enjoy minimax rate optimality in
both low dimensions and high dimensions. Experiments in regression,
classification, and neural networks show excellent performance of the proposed
methodology at the occurrence of gross outliers.
( 2
min )
In this work we establish an algorithm and distribution independent
non-asymptotic trade-off between the model size, excess test loss, and training
loss of linear predictors. Specifically, we show that models that perform well
on the test data (have low excess loss) are either "classical" -- have training
loss close to the noise level, or are "modern" -- have a much larger number of
parameters compared to the minimum needed to fit the training data exactly.
We also provide a more precise asymptotic analysis when the limiting spectral
distribution of the whitened features is Marchenko-Pastur. Remarkably, while
the Marchenko-Pastur analysis is far more precise near the interpolation peak,
where the number of parameters is just enough to fit the training data, it
coincides exactly with the distribution independent bound as the level of
overparametrization increases.
( 2
min )
Precise estimation of cross-correlation or similarity between two random
variables lies at the heart of signal detection, hyperdimensional computing,
associative memories, and neural networks. Although a vast literature exists on
different methods for estimating cross-correlations, the question what is the
best and simplest method to estimate cross-correlations using finite samples ?
is still not clear. In this paper, we first argue that the standard empirical
approach might not be the optimal method even though the estimator exhibits
uniform convergence to the true cross-correlation. Instead, we show that there
exists a large class of simple non-linear functions that can be used to
construct cross-correlators with a higher signal-to-noise ratio (SNR). To
demonstrate this, we first present a general mathematical framework using
Price's Theorem that allows us to analyze cross-correlators constructed using a
mixture of piece-wise linear functions. Using this framework and
high-dimensional embedding, we show that some of the most promising
cross-correlators are based on Huber's loss functions, margin-propagation (MP)
functions, and the log-sum-exp functions.
( 2
min )
We study the training dynamics of shallow neural networks, in a two-timescale
regime in which the stepsizes for the inner layer are much smaller than those
for the outer layer. In this regime, we prove convergence of the gradient flow
to a global optimum of the non-convex optimization problem in a simple
univariate setting. The number of neurons need not be asymptotically large for
our result to hold, distinguishing our result from popular recent approaches
such as the neural tangent kernel or mean-field regimes. Experimental
illustration is provided, showing that the stochastic gradient descent behaves
according to our description of the gradient flow and thus converges to a
global optimum in the two-timescale regime, but can fail outside of this
regime.
( 2
min )
We extend the global convergence result of Chatterjee
\cite{chatterjee2022convergence} by considering the stochastic gradient descent
(SGD) for non-convex objective functions. With minimal additional assumptions
that can be realized by finitely wide neural networks, we prove that if we
initialize inside a local region where the \L{}ajasiewicz condition holds, with
a positive probability, the stochastic gradient iterates converge to a global
minimum inside this region. A key component of our proof is to ensure that the
whole trajectories of SGD stay inside the local region with a positive
probability. For that, we assume the SGD noise scales with the objective
function, which is called machine learning noise and achievable in many real
examples. Furthermore, we provide a negative argument to show why using the
boundedness of noise with Robbins-Monro type step sizes is not enough to keep
the key component valid.
( 2
min )
submitted by /u/LiveFromChabougamou
[link] [comments]
( 42
min )
Repo: https://github.com/h2oai/h2ogpt
From the repo:
- Open-source repository with fully permissive, commercially usable code, data and models
- Code for preparing large open-source datasets as instruction datasets for fine-tuning of large language models (LLMs), including prompt engineering
- Code for fine-tuning large language models (currently up to 20B parameters) on commodity hardware and enterprise GPU servers (single or multi node)
- Code to run a chatbot on a GPU server, with shareable end-point with Python client API
- Code to evaluate and compare the performance of fine-tuned LLMs
submitted by /u/luizluiz
[link] [comments]
( 43
min )
Code & Demo: https://github.com/z-x-yang/Segment-and-Track-Anything
https://reddit.com/link/12rne1j/video/kepu2xsg9tua1/player
WebUI App is also available
https://preview.redd.it/s8uub4ii9tua1.png?width=1371&format=png&auto=webp&s=0bc91232439543fe911679d0df5fb27565b56a77
submitted by /u/liulei-li
[link] [comments]
( 43
min )
The ability to effectively handle and process enormous amounts of documents has become essential for enterprises in the modern world. Due to the continuous influx of information that all enterprises deal with, manually classifying documents is no longer a viable option. Document classification models can automate the procedure and help organizations save time and resources. […]
( 10
min )
Businesses are increasingly using machine learning (ML) to make near-real-time decisions, such as placing an ad, assigning a driver, recommending a product, or even dynamically pricing products and services. ML models make predictions given a set of input data known as features, and data scientists easily spend more than 60% of their time designing and […]
( 15
min )
This is a guest post co-written with Fred Wu from Sportradar. Sportradar is the world’s leading sports technology company, at the intersection between sports, media, and betting. More than 1,700 sports federations, media outlets, betting operators, and consumer platforms across 120 countries rely on Sportradar knowhow and technology to boost their business. Sportradar uses data […]
( 10
min )
MIT researchers exhibit a new advancement in autonomous drone navigation, using brain-inspired liquid neural networks that excel in out-of-distribution scenarios.
( 9
min )
Shanghai is once again showing why it’s called the “Magic City” as more than 1,000 exhibitors from 20 countries dazzle the automotive world this week at the highly anticipated International Automobile Industry Exhibition. With nearly 1,500 vehicles on display, the 20th edition of Auto Shanghai is showcasing the newest AI-powered cars and mobility solutions using Read article >
( 8
min )
This week’s In the NVIDIA Studio artists specializing in 3D, Gianluca Squillace and Pasquale Scionti, benefitted from just that — in their individual work and in collaborating to construct the final scene for their project, Cold Inside Diorama.
( 7
min )
For many people, opening door handles or moving a pen between their fingers is a movement that happens multiple times a day, often without much thought. For a robot, however, these movements aren’t always so easy. In reinforcement learning, robots learn to perform tasks by exploring their environments, receiving signals along the way that indicate […]
The post Unifying learning from preferences and demonstration via a ranking game for imitation learning appeared first on Microsoft Research.
( 15
min )
I develop a simplest traffic simulator of have five cars, I want improve the ability of cars's dirve using basic reinforcement learning skill.
I used tkinter to render and display the maps, But I found that tkinter can't support maps that have more than 20 row and columns in my person machine(Mac M1 mini), I don't know how to display bigger maps that have more rows and columns.
I'm very grateful that if you have some suggestion.
github repositories: https://github.com/wa008/reinforcement-learning
submitted by /u/waa007
[link] [comments]
( 42
min )
In this paper, we introduce four main novelties: First, we present a new way
of handling the topology problem of normalizing flows. Second, we describe a
technique to enforce certain classes of boundary conditions onto normalizing
flows. Third, we introduce the I-Spline bijection, which, similar to previous
work, leverages splines but, in contrast to those works, can be made
arbitrarily often differentiable. And finally, we use these techniques to
create Waveflow, an Ansatz for the one-space-dimensional multi-particle
fermionic wave functions in real space based on normalizing flows, that can be
efficiently trained with Variational Quantum Monte Carlo without the need for
MCMC nor estimation of a normalization constant. To enforce the necessary
anti-symmetry of fermionic wave functions, we train the normalizing flow only
on the fundamental domain of the permutation group, which effectively reduces
it to a boundary value problem.
( 2
min )
The article reviews significant advances in networked signal and information
processing, which have enabled in the last 25 years extending decision making
and inference, optimization, control, and learning to the increasingly
ubiquitous environments of distributed agents. As these interacting agents
cooperate, new collective behaviors emerge from local decisions and actions.
Moreover, and significantly, theory and applications show that networked
agents, through cooperation and sharing, are able to match the performance of
cloud or federated solutions, while offering the potential for improved
privacy, increasing resilience, and saving resources.
( 2
min )
This paper proposes a novel centralized training and distributed execution
(CTDE)-based multi-agent deep reinforcement learning (MADRL) method for
multiple unmanned aerial vehicles (UAVs) control in autonomous mobile access
applications. For the purpose, a single neural network is utilized in
centralized training for cooperation among multiple agents while maximizing the
total quality of service (QoS) in mobile access applications.
( 2
min )
Consumer's privacy is a main concern in Smart Grids (SGs) due to the
sensitivity of energy data, particularly when used to train machine learning
models for different services. These data-driven models often require huge
amounts of data to achieve acceptable performance leading in most cases to
risks of privacy leakage. By pushing the training to the edge, Federated
Learning (FL) offers a good compromise between privacy preservation and the
predictive performance of these models. The current paper presents an overview
of FL applications in SGs while discussing their advantages and drawbacks,
mainly in load forecasting, electric vehicles, fault diagnoses, load
disaggregation and renewable energies. In addition, an analysis of main design
trends and possible taxonomies is provided considering data partitioning, the
communication topology, and security mechanisms. Towards the end, an overview
of main challenges facing this technology and potential future directions is
presented.
( 2
min )
This paper presents the approach and results of USC SAIL's submission to the
Signal Processing Grand Challenge 2023 - e-Prevention (Task 2), on detecting
relapses in psychotic patients. Relapse prediction has proven to be
challenging, primarily due to the heterogeneity of symptoms and responses to
treatment between individuals. We address these challenges by investigating the
use of sleep behavior features to estimate relapse days as outliers in an
unsupervised machine learning setting. We extract informative features from
human activity and heart rate data collected in the wild, and evaluate various
combinations of feature types and time resolutions. We found that short-time
sleep behavior features outperformed their awake counterparts and larger time
intervals. Our submission was ranked 3rd in the Task's official leaderboard,
demonstrating the potential of such features as an objective and non-invasive
predictor of psychotic relapses.
( 2
min )
Fetal standard scan plane detection during 2-D mid-pregnancy examinations is
a highly complex task, which requires extensive medical knowledge and years of
training. Although deep neural networks (DNN) can assist inexperienced
operators in these tasks, their lack of transparency and interpretability limit
their application. Despite some researchers have been committed to visualizing
the decision process of DNN, most of them only focus on the pixel-level
features and do not take into account the medical prior knowledge. In this
work, we propose an interpretable framework based on key medical concepts,
which provides explanations from the perspective of clinicians' cognition.
Moreover, we utilize a concept-based graph convolutional neural(GCN) network to
construct the relationships between key medical concepts. Extensive
experimental analysis on a private dataset has shown that the proposed method
provides easy-to-understand insights about reasoning results for clinicians.
( 2
min )
Self-supervised monocular depth estimation approaches suffer not only from
scale ambiguity but also infer temporally inconsistent depth maps w.r.t. scale.
While disambiguating scale during training is not possible without some kind of
ground truth supervision, having scale consistent depth predictions would make
it possible to calculate scale once during inference as a post-processing step
and use it over-time. With this as a goal, a set of temporal consistency losses
that minimize pose inconsistencies over time are introduced. Evaluations show
that introducing these constraints not only reduces depth inconsistencies but
also improves the baseline performance of depth and ego-motion prediction.
( 2
min )
In this paper, we primarily focus on understanding the data preprocessing
pipeline for DNN Training in the public cloud. First, we run experiments to
test the performance implications of the two major data preprocessing methods
using either raw data or record files. The preliminary results show that data
preprocessing is a clear bottleneck, even with the most efficient software and
hardware configuration enabled by NVIDIA DALI, a high-optimized data
preprocessing library. Second, we identify the potential causes, exercise a
variety of optimization methods, and present their pros and cons. We hope this
work will shed light on the new co-design of ``data storage, loading pipeline''
and ``training framework'' and flexible resource configurations between them so
that the resources can be fully exploited and performance can be maximized.
( 2
min )
In this paper, we extends original Neural Collapse Phenomenon by proving
Generalized Neural Collapse hypothesis. We obtain Grassmannian Frame structure
from the optimization and generalization of classification. This structure
maximally separates features of every two classes on a sphere and does not
require a larger feature dimension than the number of classes. Out of curiosity
about the symmetry of Grassmannian Frame, we conduct experiments to explore if
models with different Grassmannian Frames have different performance. As a
result, we discover the Symmetric Generalization phenomenon. We provide a
theorem to explain Symmetric Generalization of permutation. However, the
question of why different directions of features can lead to such different
generalization is still open for future investigation.
( 2
min )
Robotic grasping in highly noisy environments presents complex challenges,
especially with limited prior knowledge about the scene. In particular,
identifying good grasping poses with Bayesian inference becomes difficult due
to two reasons: i) generating data from uninformative priors proves to be
inefficient, and ii) the posterior often entails a complex distribution defined
on a Riemannian manifold. In this study, we explore the use of implicit
representations to construct scene-dependent priors, thereby enabling the
application of efficient simulation-based Bayesian inference algorithms for
determining successful grasp poses in unstructured environments. Results from
both simulation and physical benchmarks showcase the high success rate and
promising potential of this approach.
( 2
min )
In this paper, we describe a method for estimating the joint probability
density from data samples by assuming that the underlying distribution can be
decomposed as a mixture of product densities with few mixture components. Prior
works have used such a decomposition to estimate the joint density from
lower-dimensional marginals, which can be estimated more reliably with the same
number of samples. We combine two key ideas: dictionaries to represent 1-D
densities, and random projections to estimate the joint distribution from 1-D
marginals, explored separately in prior work. Our algorithm benefits from
improved sample complexity over the previous dictionary-based approach by using
1-D marginals for reconstruction. We evaluate the performance of our method on
estimating synthetic probability densities and compare it with the previous
dictionary-based approach and Gaussian Mixture Models (GMMs). Our algorithm
outperforms these other approaches in all the experimental settings.
( 2
min )
This study presents a benchmark for evaluating action-constrained
reinforcement learning (RL) algorithms. In action-constrained RL, each action
taken by the learning system must comply with certain constraints. These
constraints are crucial for ensuring the feasibility and safety of actions in
real-world systems. We evaluate existing algorithms and their novel variants
across multiple robotics control environments, encompassing multiple action
constraint types. Our evaluation provides the first in-depth perspective of the
field, revealing surprising insights, including the effectiveness of a
straightforward baseline approach. The benchmark problems and associated code
utilized in our experiments are made available online at
github.com/omron-sinicx/action-constrained-RL-benchmark for further research
and development.
( 2
min )
Trained computer vision models are assumed to solve vision tasks by imitating
human behavior learned from training labels. Most efforts in recent vision
research focus on measuring the model task performance using standardized
benchmarks. Limited work has been done to understand the perceptual difference
between humans and machines. To fill this gap, our study first quantifies and
analyzes the statistical distributions of mistakes from the two sources. We
then explore human vs. machine expertise after ranking tasks by difficulty
levels. Even when humans and machines have similar overall accuracies, the
distribution of answers may vary. Leveraging the perceptual difference between
humans and machines, we empirically demonstrate a post-hoc human-machine
collaboration that outperforms humans or machines alone.
( 2
min )
We present LTC-SE, an improved version of the Liquid Time-Constant (LTC)
neural network algorithm originally proposed by Hasani et al. in 2021. This
algorithm unifies the Leaky-Integrate-and-Fire (LIF) spiking neural network
model with Continuous-Time Recurrent Neural Networks (CTRNNs), Neural Ordinary
Differential Equations (NODEs), and bespoke Gated Recurrent Units (GRUs). The
enhancements in LTC-SE focus on augmenting flexibility, compatibility, and code
organization, targeting the unique constraints of embedded systems with limited
computational resources and strict performance requirements. The updated code
serves as a consolidated class library compatible with TensorFlow 2.x, offering
comprehensive configuration options for LTCCell, CTRNN, NODE, and CTGRU
classes. We evaluate LTC-SE against its predecessors, showcasing the advantages
of our optimizations in user experience, Keras function compatibility, and code
clarity. These refinements expand the applicability of liquid neural networks
in diverse machine learning tasks, such as robotics, causality analysis, and
time-series prediction, and build on the foundational work of Hasani et al.
( 2
min )
Modern deep models for summarization attains impressive benchmark
performance, but they are prone to generating miscalibrated predictive
uncertainty. This means that they assign high confidence to low-quality
predictions, leading to compromised reliability and trustworthiness in
real-world applications. Probabilistic deep learning methods are common
solutions to the miscalibration problem. However, their relative effectiveness
in complex autoregressive summarization tasks are not well-understood. In this
work, we thoroughly investigate different state-of-the-art probabilistic
methods' effectiveness in improving the uncertainty quality of the neural
summarization models, across three large-scale benchmarks with varying
difficulty. We show that the probabilistic methods consistently improve the
model's generation and uncertainty quality, leading to improved selective
generation performance (i.e., abstaining from low-quality summaries) in
practice. We also reveal notable failure patterns of probabilistic methods
widely-adopted in NLP community (e.g., Deep Ensemble and Monte Carlo Dropout),
cautioning the importance of choosing appropriate method for the data setting.
( 2
min )
In this paper we study a class of constrained minimax problems. In
particular, we propose a first-order augmented Lagrangian method for solving
them, whose subproblems turn out to be a much simpler structured minimax
problem and are suitably solved by a first-order method recently developed in
[26] by the authors. Under some suitable assumptions, an \emph{operation
complexity} of ${\cal O}(\varepsilon^{-4}\log\varepsilon^{-1})$, measured by
its fundamental operations, is established for the first-order augmented
Lagrangian method for finding an $\varepsilon$-KKT solution of the constrained
minimax problems.
( 2
min )
The Linear-Quadratic Regulation (LQR) problem with unknown system parameters
has been widely studied, but it has remained unclear whether $\tilde{
\mathcal{O}}(\sqrt{T})$ regret, which is the best known dependence on time, can
be achieved almost surely. In this paper, we propose an adaptive LQR controller
with almost surely $\tilde{ \mathcal{O}}(\sqrt{T})$ regret upper bound. The
controller features a circuit-breaking mechanism, which circumvents potential
safety breach and guarantees the convergence of the system parameter estimate,
but is shown to be triggered only finitely often and hence has negligible
effect on the asymptotic performance of the controller. The proposed controller
is also validated via simulation on Tennessee Eastman Process~(TEP), a commonly
used industrial process example.
( 2
min )
In this paper, a critical bibliometric analysis study is conducted, coupled
with an extensive literature survey on recent developments and associated
applications in machine learning research with a perspective on Africa. The
presented bibliometric analysis study consists of 2761 machine learning-related
documents, of which 98% were articles with at least 482 citations published in
903 journals during the past 30 years. Furthermore, the collated documents were
retrieved from the Science Citation Index EXPANDED, comprising research
publications from 54 African countries between 1993 and 2021. The bibliometric
study shows the visualization of the current landscape and future trends in
machine learning research and its application to facilitate future
collaborative research and knowledge exchange among authors from different
research institutions scattered across the African continent.
( 2
min )
Gaussian processes are arguably the most important class of spatiotemporal
models within machine learning. They encode prior information about the modeled
function and can be used for exact or approximate Bayesian learning. In many
applications, particularly in physical sciences and engineering, but also in
areas such as geostatistics and neuroscience, invariance to symmetries is one
of the most fundamental forms of prior information one can consider. The
invariance of a Gaussian process' covariance to such symmetries gives rise to
the most natural generalization of the concept of stationarity to such spaces.
In this work, we develop constructive and practical techniques for building
stationary Gaussian processes on a very large class of non-Euclidean spaces
arising in the context of symmetries. Our techniques make it possible to (i)
calculate covariance kernels and (ii) sample from prior and posterior Gaussian
processes defined on such spaces, both in a practical manner. This work is
split into two parts, each involving different technical considerations: part I
studies compact spaces, while part II studies non-compact spaces possessing
certain structure. Our contributions make the non-Euclidean Gaussian process
models we study compatible with well-understood computational techniques
available in standard Gaussian process software packages, thereby making them
accessible to practitioners.
( 2
min )
Chen et al. [Chen2022] recently published the article 'Fast and scalable
search of whole-slide images via self-supervised deep learning' in Nature
Biomedical Engineering. The authors call their method 'self-supervised image
search for histology', short SISH. We express our concerns that SISH is an
incremental modification of Yottixel, has used MinMax binarization but does not
cite the original works, and is based on a misnomer 'self-supervised image
search'. As well, we point to several other concerns regarding experiments and
comparisons performed by Chen et al.
( 2
min )
Adaptation-relevant predictions of climate change are often derived by
combining climate model simulations in a multi-model ensemble. Model evaluation
methods used in performance-based ensemble weighting schemes have limitations
in the context of high-impact extreme events. We introduce a locally
time-invariant method for evaluating climate model simulations with a focus on
assessing the simulation of extremes. We explore the behaviour of the proposed
method in predicting extreme heat days in Nairobi and provide comparative
results for eight additional cities.
( 2
min )
Enabling resilient autonomous motion planning requires robust predictions of
surrounding road users' future behavior. In response to this need and the
associated challenges, we introduce our model titled MTP-GO. The model encodes
the scene using temporal graph neural networks to produce the inputs to an
underlying motion model. The motion model is implemented using neural ordinary
differential equations where the state-transition functions are learned with
the rest of the model. Multimodal probabilistic predictions are obtained by
combining the concept of mixture density networks and Kalman filtering. The
results illustrate the predictive capabilities of the proposed model across
various data sets, outperforming several state-of-the-art methods on a number
of metrics.
( 2
min )
Nowadays, face recognition systems surpass human performance on several
datasets. However, there are still edge cases that the machine can't correctly
classify. This paper investigates the effect of a combination of machine and
human operators in the face verification task. First, we look closer at the
edge cases for several state-of-the-art models to discover common datasets'
challenging settings. Then, we conduct a study with 60 participants on these
selected tasks with humans and provide an extensive analysis. Finally, we
demonstrate that combining machine and human decisions can further improve the
performance of state-of-the-art face verification systems on various benchmark
datasets. Code and data are publicly available on GitHub.
( 2
min )
Stochastic gradient Langevin dynamics (SGLD) are a useful methodology for
sampling from probability distributions. This paper provides a finite sample
analysis of a passive stochastic gradient Langevin dynamics algorithm (PSGLD)
designed to achieve inverse reinforcement learning. By "passive", we mean that
the noisy gradients available to the PSGLD algorithm (inverse learning process)
are evaluated at randomly chosen points by an external stochastic gradient
algorithm (forward learner). The PSGLD algorithm thus acts as a randomized
sampler which recovers the cost function being optimized by this external
process. Previous work has analyzed the asymptotic performance of this passive
algorithm using stochastic approximation techniques; in this work we analyze
the non-asymptotic performance. Specifically, we provide finite-time bounds on
the 2-Wasserstein distance between the passive algorithm and its stationary
measure, from which the reconstructed cost function is obtained.
( 2
min )
In this paper we study a class of constrained minimax problems. In
particular, we propose a first-order augmented Lagrangian method for solving
them, whose subproblems turn out to be a much simpler structured minimax
problem and are suitably solved by a first-order method recently developed in
[26] by the authors. Under some suitable assumptions, an \emph{operation
complexity} of ${\cal O}(\varepsilon^{-4}\log\varepsilon^{-1})$, measured by
its fundamental operations, is established for the first-order augmented
Lagrangian method for finding an $\varepsilon$-KKT solution of the constrained
minimax problems.
( 2
min )
There is an increasing interest in the development of new data-driven models
useful to assess the performance of communication networks. For many
applications, like network monitoring and troubleshooting, a data model is of
little use if it cannot be interpreted by a human operator. In this paper, we
present an extension of the Multivariate Big Data Analysis (MBDA) methodology,
a recently proposed interpretable data analysis tool. In this extension, we
propose a solution to the automatic derivation of features, a cornerstone step
for the application of MBDA when the amount of data is massive. The resulting
network monitoring approach allows us to detect and diagnose disparate network
anomalies, with a data-analysis workflow that combines the advantages of
interpretable and interactive models with the power of parallel processing. We
apply the extended MBDA to two case studies: UGR'16, a benchmark flow-based
real-traffic dataset for anomaly detection, and Dartmouth'18, the longest and
largest Wi-Fi trace known to date.
( 2
min )
Deep metric learning techniques have been used for visual representation in
various supervised and unsupervised learning tasks through learning embeddings
of samples with deep networks. However, classic approaches, which employ a
fixed distance metric as a similarity function between two embeddings, may lead
to suboptimal performance for capturing the complex data distribution. The
Bregman divergence generalizes measures of various distance metrics and arises
throughout many fields of deep metric learning. In this paper, we first show
how deep metric learning loss can arise from the Bregman divergence. We then
introduce a novel method for learning empirical Bregman divergence directly
from data based on parameterizing the convex function underlying the Bregman
divergence with a deep learning setting. We further experimentally show that
our approach performs effectively on five popular public datasets compared to
other SOTA deep metric learning methods, particularly for pattern recognition
problems.
( 2
min )
Data - https://github.com/allenai/mmc4
submitted by /u/MysteryInc152
[link] [comments]
( 43
min )
submitted by /u/Phaen_
[link] [comments]
( 45
min )
Announcements ChatGPT, The Overconfident Artist Two weeks ago, I wrote about the concepts of XAI, or explainable artificial intelligence. The basic concept is XAI is artificial intelligence designed in a way that allows it to explain its decision-making process. This is especially useful for users that want to question answers given by AI chatbots, or… Read More »DSC Weekly 11 April 2023 – ChatGPT, The Overconfident Artist
The post DSC Weekly 11 April 2023 – ChatGPT, The Overconfident Artist appeared first on Data Science Central.
( 21
min )
Data warehouses are at the heart of any organization’s technology ecosystem. The emergence of cloud technology has enabled data warehouses to offer capabilities such as cost-effective data storage, scalable computing and storage, utilization-based pricing, and fully managed service delivery. As data consumption increases and more people live and work remotely, companies are adopting modern data… Read More »Why It’s Important to Change Misconceptions About Data Warehouse Technology
The post Why It’s Important to Change Misconceptions About Data Warehouse Technology appeared first on Data Science Central.
( 21
min )
Three years after the outbreak of the COVID-19 pandemic, the lingering impacts of the viral outbreak and the risk of another deadly pathogen spreading around the world remain. The pandemic challenged every health system in the world, stressing facilities, medical equipment suppliers, and medical personnel. Public health authorities tracked disease transmission, modeled forecasts across multiple… Read More »How Informatics, ML, and AI Can Better Prepare the Healthcare Industry for the Next Global Pandemic
The post How Informatics, ML, and AI Can Better Prepare the Healthcare Industry for the Next Global Pandemic appeared first on Data Science Central.
( 21
min )
Over the years of working on commercial AI projects, I have come across various products and business domains that have adopted machine learning algorithms. This experience helped me form some best practices for selecting the right algorithms for different types of business tasks. In this article, I will share some valuable insights from my experience regarding how to work with them in the most efficient way to meet the client’s business needs.
The post 3 Relevant ML Algorithms Commonly Used in Commercial AI Projects appeared first on Data Science Central.
( 24
min )
Artificial Intelligence (AI) is sweeping the globe, leaving no stone unturned as it reshapes industries far and wide.
The post Harnessing the Power of OpenAI Technology: 5 Innovative Marketing Tools appeared first on Data Science Central.
( 20
min )
Large language models (LLMs) with billions of parameters are currently at the forefront of natural language processing (NLP). These models are shaking up the field with their incredible abilities to generate text, analyze sentiment, translate languages, and much more. With access to massive amounts of data, LLMs have the potential to revolutionize the way we […]
( 18
min )
Large language models (LLMs) with billions of parameters are currently at the forefront of natural language processing (NLP). These models are shaking up the field with their incredible abilities to generate text, analyze sentiment, translate languages, and much more. With access to massive amounts of data, LLMs have the potential to revolutionize the way we […]
( 18
min )
Amazon Kendra is an intelligent search service powered by machine learning (ML), enabling organizations to provide relevant information to customers and employees, when they need it. Amazon Kendra uses ML algorithms to enable users to use natural language queries to search for information scattered across multiple data souces in an enterprise, including commonly used document […]
( 7
min )
This post was co-written with Dave Gowel, CEO of RallyPoint. In his own words, “RallyPoint is an online social and professional network for veterans, service members, family members, caregivers, and other civilian supporters of the US armed forces. With two million members on the platform, the company provides a comfortable place for this deserving population […]
( 9
min )
Reliability managers and technicians in industrial environments such as manufacturing production lines, warehouses, and industrial plants are keen to improve equipment health and uptime to maximize product output and quality. Machine and process failures are often addressed by reactive activity after incidents happen or by costly preventive maintenance, where you run the risk of over-maintaining […]
( 16
min )
In the first two blog posts in this series, we presented our vision for Cloud Intelligence/AIOps (AIOps) research, and scenarios where innovations in AI technologies can help build and operate complex cloud platforms and services effectively and efficiently at scale. In this blog post, we dive deeper into our efforts to automatically manage large-scale cloud […]
The post Automatic post-deployment management of cloud applications appeared first on Microsoft Research.
( 15
min )
Sponsored Post Attend the Data Science Symposium 2022 on November 8 The Center for Business Analytics at the University of Cincinnati will present its annual Data Science Symposium 2022 on November 8. This all day in-person event will have three featured speakers and two tech talk tracks with four concurrent presentations in each track. The […]
The post Attend the Data Science Symposium 2022, November 8 in Cincinnati appeared first on Machine Learning Mastery.
( 10
min )
